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FOREWORD 



When the U.S. Office of Education was 
chartered in 1867 , one charge to its 
commissioners was to determine the 
nation's progress in education. Hie 
National Assessment of Educational 
Pr ogress (NAEP) was initiated a cen- 
tury later to address, in a systemat- 
ic way, that charge. 

Since 1969, the National Assessment 
has gathered information about levels 
of educational achievement across the 
country and reported its findings to 
the nation. It has surveyed the at- 
tainments of 9-year-olds, 13-year- 
olds, 17-year-olds and adults in art, 
career and occupational developnent, 
citizenship, literature, mathematics, 
music, reading, science, social stud- 
ies and writing. All areas have been 
periodically reassessed in order to 
detect any important changes, To 
date, National Assessment has inter- 
viewed, and tested nearly 1 ; 000,000 
young Americans. 

Learning-area assessments evolve from 
a consensus process. Each assessment 
is the product of several years of 
work by a great many educators, 
scholars and lay persons from all 
over the nation. Initially, these 
people design objectives for each 
subject area, proposing general goals 
they feel Americans should be achiev- 



ing in the course of their education. 
After careful review, these objec- 
tives are given to writers, whose 
task is to create exercises (items) 
appropriate to the objectives. 

When the exercises have passed exten- 
sive reviews by subject-area special- 
ists, measurement experts and lay 
persons, they are administered to 
probability samples. The people in 
these samples are selected in such a 
way that the results of their assess- 
ment can be generalized to an entire 
national population. That is, on the 
basis of the performance of about 
2,500 9-year-olds on a given exer- 
cise, we can make generalizations 
about the probable performance of all 
9-year-olds in the nation. 

After assessment data have been col- 
lected, scored and analyzed, the Na- 
tional Assessment publishes reports 
and disseminates the results as 
widely as possible. Not all exercises 
are released for publication. Because 
NAEP will readminister some of the 
same exercises in the future to de- 
termine whether the performance lev- 
els of Americans have increased, re- 
mained stable oic decreased, it is 
essential that they not be released 
in order to preserve the integrity of 
the study. 
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9 



ACKNOWLEDGMENTS 



Assessing reading performance of 
young Americans throughout the nation 
is an undertaking of major propor- 
tions. Certainly it could not have 
become a reality without substantial 
contributions by many people, not the 
least of whom are the students, 
teachers and administrators who coop- 
erated so generously. 

Special thanks are due to the dozens 
of consultants — both subject-area 
specialists and lay persons — who 
reviewed the materials used in the 
three reading assessments under the 
general guidance of the National 
Assessment of Educational Progress 
(NAEP) staff. Particular acknowledg- 
ment is given to Carita Chapman, Bu- 
reau of Reading Improvement, Chicago, 
Illinois; Charles Cooper, University 
of California at San Diego; Anthony 
Petrosky, University of Pittsburgh, 
Pennsylvania; Beverly Roller, Jeffer- 
son County Public Schools, 
WheatRidge, Colorado; Robert 
Schreiner, University of Minnesota; 
and Dorothy Strickland, Columbia 
University, New York, all of whom 
provided special assistance in sev- 
eral areas of the assessment, 

Our gratitude is also extended to the 
Reading/Literature Advisory Committee 
(see Appendix A) and other reading 
experts who participated in advisory 
and interpretive conferences for the 
three reading assessments. Hie staff 
at NAEP appreciates the insights pro- 
vided by these distinguished **duca- 
tion and subject-area specialists. 



Administration of exercises was han- 
dled by the Research Triangle Insti- 
tute. Scor ing and processing we re 
performed by the Westinghouse Data- 
Score Systems (formerly the Measure- 
ment Research Center) , Iowa City, 
Iowa, and by the National Assessment 
staff, Dan Duse and Donna Benson de- 
serve special credit for their excel- 
lent work in supervising and proces- 
sing the scoring of open-ended exer- 
cises. 

Every assessment is the result of a 
collaborative effort by the National 
Assessment staff. Many persons con- 
tributed to the reading assessments. 
Special thanks are extended to Nancy 
Mead for developnent and analysis; 
Michael Noe for developnent, analysis 
and scoring; Rex Brown for develop- 
ment and editorial supervision; Kay 
Barrow for technical planning and 
analysis; Ina Mullis for scoring and 
technical guidance; Donald Searls for 
sampling and analysis guidance; 
Dunlap Scott for coordinating data 
collection; John Kalk, Suzie Sullivan 
and Gw*?n Edwards for data processing 
support; Ava Powell for technical 
support; and Marci Reser and Deborah 
Houy for report production. 



— . 

Roy H. Forbes 
Director 



9 

ERIC 



INTRODUCTION 



The National Assessment of Educa- 
tional Progress (NAEP) first assessed 
reading and literature achievement in 
separate arsessments during the 
1970-71 school year. Since that time, 
reading has beer, assessed twice as a 
discrete learning area (1974-75 and 
1979-80) and reading and literature 
have been combined for a joint 
assessment in 1979-80. Each assess- 
ment surveyed the achievement and 
attitudes of American 9-, 13- and 
17-year-olds, using a deeply strati- 
fied, multistage probability sample 
design. 

To measure changes in reading per- 
formance between 1970-71, 1974-75 and 
1979-80, approximately half of the 
exercises asse^gfed in the first 
assessment were reassessed in the 
second and third under almost identi- 
cal administrative conditions, To 
measure the status of reading/litera- 
ture achievement in 1979-80, National 
Assessment consultants developed new 
objectives and developed additional 
exercises to provide coverage of the 
new objectives. Some 1970-71 litera- 
ture items were reassessed. 

Approximately 29,000 9-year-olds, 
41,000 13-year-olds and 36,000 17- 
year-olds participated in the 1979-80 
reading assessment. Because National 
Assessment reports results for groups 
of students, not individuals, it is 
not necessary for each student to 
respond to every item (exercise) . 1 



^tional Assessment uses the term 
"exercise* to mean an assessment 



Each respondent completed only one 
item booklet of about 45 minutes in 
length. Between 2,500 and 2,900 stu- 
dents responded to each booklet. In 
1979-80 there were 11 exercise book- 
lets for 9-year-olds, 15 booklets for 
13-year-olds and 14 booklets foi 17- 
year-olds. 

The exer c i ses f o r each assessment 
were administered by a professional 
data collection staff to minimize the 
burden on participating schools and 
to maximize uniformity of assessment 
conditions. Instructions were re- 
corded on a paced audio tape and 
piayed back to students to ensure 
that all students moved through the 
packages at the same speed. 

The majority of the items were multi- 
ple-choice; a 5 few exercises were 
open-ended. Each item included one or 
more item parts. 

Multiple-choice items were scored by 
an optical scanning machine; open- 
ended items were hand-scored by 
trained scorers using scoring guides 
developed to define categories of 
acceptable and unacceptable respons- 
es. 

National Assessment reports estimated 
percentages of correct responses for 
single items. When a report indicates 
that "85% of the 17-year-olds gave a 
correct response," it means that an 



item. The terms "exercise" and "item" 
are used interchangeably in this re- 
port. 
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estimated 85% of the 17-year-olds 
would have given a correct response 
if all the 17-year-olds in schools 
across the country had been assessed* 
In addition to reporting national 
results, National Assessment provides 
data on the performance of various 
population subgroups within the na- 
tional population, defined by sex, 
race, region of the country, size and 
type of community lived in and level 
of parental education. National 
Assessment also aggregates percent- 
ages of success on various sets of 
items to provide data on changes in 
performance between assessments and 
on the differential performance of 
population subgroups. 

This handbook describes the 
procedures used to develop, 
administer and analyze the results of 
the 1979-80 reading/literature 
assessment. In doing so, it will 
touch briefly on the characteristics 
of earlier reading and literature 
assessments and will describe changes 
in goals and methods employed in each 
assessment. The primary purpose of 



this handbook is to provide detailed 
procedural information for people 
interested in re'plicating the 
assessment or in need of more 
information than is provided in the 
reports containing assessment data. 
Ttie eight chapters cover objectives 
redevelopment, exercise creation, 
preparation of assessment booklets, 
sampling, data collection, scoring, 
data analysis and reporting , Each 
chapter explains the basic procedures 
used for the 1979-80 assessment, 
contrasts the procedures to those 
used in earlier years (if there were 
changes) and discusses relevant 
theoretical and practical issues. 



Appendicular matGrials cover 
definitions of reporting groups, 
forms used to gather background 
info rmat ion about students and 
schools, response rates, computation 
of achievement measures and 
procedures for smoothing respondent 
weights. A glossary of National 
Assessment terms is provided at the 
end of the book. 
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NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 

Errata Sheet tot Report li— RL— 40 
Procedural Handbook: 
1979-80 Reading and Literature Assessment 
September '981 



Page 15 The last complete sentence in the left-hand 
column should read: 

Respondents were given the opportunity to 
complete up to three assessment booklets and 
were remunerated at the rate of $5. 00 each 
for one'or two completed booklet*, or $20.00 
for three booklets. 

Page 20 Right-hand column, second full paragraph, 
last sentence, "Qualitative ranks we re 
assigned. . .information was obtained." This 
sentence should be moved to the end of the 
next paragraph, following the words 
"supporting the emotion." 

P^ge 26 A footnote reference should be added to the 
last word — "impact. " The footnote should 
read: 

5 Fourteen exercises included in the 1970-71 
and 1974-75 assessments were not administered 
in the 1979-80 assessment as the stimulus 
materials did not meet current standards of 
freedom from bias and stereotyping. The 
summary measures were recomputed to exclude 
these exercises from the previously repc "ed 
results for 1970-71 and 1974-75. 

Page 78 Table E-2* second to last line — "<4, <8, 
12" — should read: 

">4, >8, 12" 
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CHAPTER 1 



OBJECTIVES REDEVELOPMENT 



The primary goal of t„e National 
Assessment of Educational Progress 
(NAEP) is to report on the current 
educational status of young Americans 
and to monitor any changes in 
achievement over time. For ea^h 
learning ar ja to be assessed, NAEP 
asks consultants to develop objec- 
tives that define the subject area. 
Since the objectives provide guide- 
lines for exercise developers, con- 
sultants are asked to include exam- 
ples of the knowledge, skiJls and 
attitudes to be assessed at erj* age 
level. 

Education in America is a 
collaborative enterprise involving a 
great many people with widely 
differing philosophies. Providing 
information about education 
nationwide would be considerably 
easier if there were consensus about 
the means and ends of American 
education, but the fact is that 
Americans have conflicting and 
sometimes contradictory values 
regarding the goals c ^ation and 
the means for achie them. To 

develop an assessment that is truly 
national in scope and takes into 
account the diversity of curricula, 
values and goals across the country, 
National Assessment employs a 
consensus process for developing 
objectives, with representation of 
many different groups of people. 

Several types of consultants help to 
develop National Assessment 
objectives. College and university 



specialists in a learning area ensure 
that the objectives include important 
concepts that the schools should be 
teach ing . Educators , includ ing 
clc ssroom teachers, curriculum 
supervisors and persons involved in 
teacher education, make sure that the 
objectives describe concepts, skills 
and attitudes that the schools should 
be teaching and those that they 
presently are teaching. Concerned 
citizens, parents and other 
interested lay persons must agree 
that the objectives are important for 
young people to achieve, are free of 
educational jargon and are not biased 
or offensive to any groups. 
Consultants are selected to represent 
different regions of the country and 
minority groups They also represent 
a range of experience with students 
of different ages and community 
types. 

When the reading and literature areas 
were merged into a single assessment, 
two sets of prior objectives existed 
for each (see Tables 1 and 2) . The 
first reading objectives were written 
in the late 1960s under the direction 
of Science Research Associates. 
Comprehensive in scope and very 
detailed, they addressed literary as 
•well as nonliterary texts, literary 
terms and skills as well as terms and 
s,. A lls more closely associated with 
reading instruction (Table 1). In 
contrast, the second reading 
objectives, written some years later, 
were much less detailed and 
concentrated solely upon the goals of 
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TABLE 1. Reading Objectives Outlined 



Objectives tor 1970-71 
Reading Assessment' 

Comprehend what is read 
A. Rraa individual words 
8. Kfad phrases, clauses, sentences 
C Read paragraphs, passages and longer 
v/orks 

Analyse what is read 
A. Be able to trace sequences 
3 Perceive the structure and organiza- 
tion of tfie work 

See the techniques by which the author 
has created his effects 

jse what is read 

* Reenter sijrpficant parts of what is 
reac 

0 Fn 11 ow wr 1 1 1 en d l r ec t lens 

1 Jbtain information efficiently 

reason logically from wr.at is read 
A ^raw ayoropnate inferences fro*; the 
.dtet^l tnat is rpad and "reaJ be- 
tween the 1 >nes *nere necessary 
5 Arrive at a qenerdl prinnple after 
examining a b^nes of devils 
feasor, from a qpneral principle to 
specific instances 

*3ake judgments concerning what is read 
A fe'ate what is read to thinqs other 

tha r i the r pacific materia 1 beinj 

red 

b Finj dnd 'jse appropriate criteria in 
f.'d, i no judgments about what li real 
Make judgrtentS aL.Qjt d wur* on tin* 
basis of what is tound in the word- 
itself 



Objectives for 1974-7 
Reading Assessment 

Demonstrate oehavior conducive to read- 
ing 

A Demonstrate values related to read- 
ing 

B Assess the readability of materials 
C Demonstrate knowl edqe of their own 
reading ai 1 1 1 ty 

Demonstrate word identification skills 
A 
B 



Know the letters of the alphabet 
Apply knowledge of sound synbol 
relationships 

Apply structural analysis techniques 

Possess basic sight vocabulary 

Use context for word identification 

spss shlls for reading comprehension 
"tilue written language conventions 
as comprehension aids 
Demonstrate literal understanding of 
naferial read 

Demonstrate inferential understanding 
of material read 



'.5P a variety of approaches in gathering 
^nfo nation 

A Demonstrate flexibility in adapting 
their rate of reading to suit thei^ 
purpose(s) and the nature of the ma- 
*eria 1 

R Possess reading study ^ills 

C Use rpfeience Materials efficiently 



Objectives for 1979-80 
Reading/Literature Assessment' 

I Values reading and literature 

• u.ut-v vt.t uinei n,i u« redutny fur 

the individual 
8. Appreciates the cultural role of 
written discourse as a way of trans- 
mitting, sustaining and changing 
the values of a soc lety 

II Comprehends written worts 

A Comprehends words and lexical rela- 
tionships 

B Comprehends propos 1 1 iona I relation- 
ships 

I. Comprehends textual relationships 

III Responds to written works in interpre- 
tive and evaluative wa,s 
A fxtends understanding of written 

works through interpretation 
r Lvaluates written works 

H Applies study skills in readinq 

A Obtains information from nonpros 

reading *at 1 1 1 tators 
B Uses the various parts of a boor 
I Obtains informal ton from mater la 1 s 

commonly found in libraries or 

re sour ce renter Z 
•1 l»se« V^n^ui st^dy techniques 
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•■ave attitudes about and an *n*erpst in 
r^a 1 1 ng 

'Jcvth of interest in reading 
l. Motives for reading 

/uaniUative measjres and rea img 
interest 



iq?n 



1974 
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TABI£ 2. Literature Objectives Outlined 



Objectives for 1970-71 
Literature Assessment 1 

I. Read literature of excellence 

A. Be acquainted with a wide variety of 
1 i terary works 

themes through which man has ex- 
pressed his values and tensions in 
Western culture 

II. Become engaged in. find meaning in and 
evaluate a work of literature 
A. Respond to a work of literature 
8. Find meanings in a work of litera- 
ture 

C. Evaluate a work of literature 

III. Develop a continuing interest and 
participation in literature and the 
literary experience 

A. Be intellectually oriented to litera- 
ture 

8. Be affectively oriented to litera- 
ture 

C Be independently active and curious 

about literature 
0. Relate literary experience to one's 

, f fe 



Objectives for Second 
Literature Assessment 2 (Cancelled) 

I Experiences literature -- is aware that 
literary qualities exist in a variety of 
forms. Seeks experiences with litera- 

tuic ill afij iCrfii t 1 «*Oiii aiijr Cui *.ui~e 

A. Listens to literature 
3. Reads literature 
. Witnesses literature 

II. Responds to literature — responds to 

literature in any form* from any culture* 
in a variety of ways — emotional ly, re- 
flectively, creatively -- and shares re- 
sponses with others 

A. Responds emotionally *- participates 
emotionally in the world of a work 
of literature 

B. Responds reflectively understands 
a work of literature by reflecting 
upon it in a variety of ways 

C. Responds creatively »- uses language 
imaginatively in response to a work 
of literature 

D. Shares responses with others -- shares 
emotional, reflective and creative 
responses in a variety of ways 

III Values literature recognizes that 

literature plays a significant continuing 
role in the experience of the individual 
and society 

A. Recognizes that literature may be a 
source of enjoyment 

B. Recognizes that experience with lit- 
erature may be a means of developing 
self-understanding and personal 

val ues 

C. Recognizes that experience wi th lit- 
erature may be a means of understand- 
ing the nature of man and the di- 
versity of cul ture 

D. Recognizes that literature and jOciety 
may Influence each other 

E. Recognizes that literature may be a 
significant means of transmitting and 
sustaining the values of a culture 



Objectives for 1979-80 
Reading/Literature Assessment 3 

I. * Val ues reading and literature 

A. Values the benefits of 'reading for 

the individual 
o A" r TCC*i*c * hn cul f y^l rclc cf 

written discourse as a way of trans 

mitting, sustaining and changing 

the values of a society 

II. Comprehends wrUten woj^s 

A Comprehend! woroVlTnd lexical rela- 
tionships ^ 

C. Comprehends proposi tional relation- 
ships 

C. Comprehends textual relationships 

III Respo/ids to written works in interpre- 
tive and evaluative ways 

A. Extends understanding of written 
works through interpretation 

B. Evaluates written works 

IV. Applies study skills in reading 

A. Obtains information from nonprose 
reading facilitators 

B. „ Uses the various parts of a book 

C. " Obtains information from materials 

commonly found in libraries or re- 
sou r ce centers 
D Uses various study techniques 



x literature CbjeatiwB, 1970. 

3 Literature itjettives, Second Assess^- nr t 1975 

***aadi*Li wd Literature it'Jf*itiu*ti t IB^S-JO Asarsfiv* 1980. 
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reading instruction defined quite 
narrowly. Consultants felt that 
reading should be differentiated from 
literature since each was a separate 
assessment area and a separate 
instructional field in the schools. 
The second objectives were also 
somewnat more o^havi orally oriented 
and more directly tied to what miqht 
be measurable. 

The first 1 iterature objectives, 
developed by Educational Testing 
Service, were keyed to activities in 
typical English courses (Table 2). 
They stressed knowledge of classic 
works, skills necessary for 
interpreting v*>rks and activities 
that promote involvement with 
literary experience. They ignored 
skills involved in learning to read. 
Ttie objectives developed for the 
1975-7*S literature assessment (which, 
for financial reasons, never took 
place) were quite different. 
Pejectinq the notions that 
"literature of excellence" could be 
defined or that acquaintance with 
classics could be assessed 
meaninqful ly, the consultants placed 
more eff.ohasis on response and 
valuing. Instead of defining 
literature as "great books," they 
defined it as "language used 
imaginatively" and created objectives 
designed to determine how much 
exposure students have had to 
imaginative language in a nunber of 
social and academic contexts. Again, 
the objectives* made no mention of 
reading skills per se . 

These were the four documents that 
served as a base for developing the 
1979-80 reading/literature assessment 
objectives. Ttie advisory committee 
charged with doing this felt that, 
although there was some overlap 
between the goals of reading and 
literature instruction, there were 
also ^reas unique to each. 



Consequently, they negotiated general 
objectives that incorporated major 
features of the previous sets but 
dropped subobjectives that no longer 
fit or had proven extremely difficult 
or impossible to assess. 

Tne objectives for the 1979-80 
assessment are categorized into four 
major areas: (1) Values Reading and 
Literature, (2) Comprehends Written 
Works, (3) Responds to Written Wbrks 
and (4) Applies Study Skills in 
Read ing . itie obj ect ives i n these 
areas have been changed somewhat from 
previous objectives in reading and 
literature, as well as merged. The 
changes are most evident in the area 
of comprehension. 

Hie comprehension objective deals 
with understanding the important 
meanings of written materials. 
Researchers and educators are still 
uncertain about how individuals 
derive meaning from the printed word. 
However, most agree that the process 
goes beyond the simple decoding of 
print into a message, it appears that 
it is a much more interactive process 
in which the reader brings his or her 
other knowledge and experience to 
bear in the reading task. Ttiis is 
evident, for example, in the various 
interpretations that different 
readers may derive from the same 
material. It is also seen in the 
activity of "reading between the 
lines." It seems clear, therefore, 
that meaning is not in the print 
alone. 

The comprehension objective reflects 
this view of reading by focusing on 
the understanding of the important 
meanings that may be derived from 
written wrks; these meanings are 
described in terms of propositions. A 
proposition is created through the 
interaction of the reader and the 
printed material. To understand 
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propositions , the reader must 
recognize the relationships between 
the concepts or ideas that can be 
derived from the printed material and 
use these relationships in 
conjunction with his or her own 
background, knowledae and experience. 



The major objective concerned with 
response to written works differs 
from the comprehension objective in 
that the term "comprehension," in the 
latter, is limited to the largely 
'unconscious activity of understanding 
pr imary meaning in the process of 
reading. Cbnscious, deliberate, overt 
attempts to extend understanding 
after something has been read — for 
example, analysis or interpretation 
of a poem, class discussion, critical 
essays, and so forth ~ are defined 
as response to written works. 
Implicit in the definition of the 
response objective is the suggestion 
that a highly desirable consequence 
of reading* is action. Much that has 
been written is intended not only to 



increase understanding but to provoke 
deliberate responses. 

The remaining two major objectives — 
Values Reading and Li terature, 
Applies Study Skills in Reading — 
have not ~hangpd mitrh from objectives 
of previous assessments. 

There was considerable di scussion 
among advisory comm i ttee member s 
about the desirability of objectives 
dealing directly with word-attack 
skills, reading rate, and skimming 
and scanning ski lis. Al though 
acknowledging lhat word-attack skills 
are important enaL'.ing skills for 
readers, a majority of the committee 
firmly decided that they are not 
outcomes of reading and so should not 
appear as specific objectives* After 
examining previous naEP assessments 
of reading rate and skimming and 
scanning and experimenting with new 
measures, the committee decided to 
assess these skills in a limited way, 
and to include them as part of the 
study skills objectives. 



ERLC 



CHAPTER 2 
DEVELOPMENT OF EXERCISES 



The Exercise 
Development Framework 

Assessments cannot be developed di- 
rectly from objectives. There are a 
number of steps in between, in read- 
ing and literature, there were deci- 
sions to be made about the kinds of 
texts that would be used, the rela- 
tive importance of certain objectives 
and the relevance of materials previ- 
ously assessed to the new objectives. 
In addition, clarity about the theo- 
retical underpinnings upon which the 
assessment, or parts of it, would 
rest was critical to ensure that ex- 
ercises would provide information 
interpretable within particular con- 
structs of the reading process. 

First, in order to assure changes in 
performance over time in the 1979-80 
assessment, it was decided that three 
booklets of items for each age group 
would consist of previously assessed 
items. These items measured perform- 
ance on literal comprehension, infer- 



ential comprehension and reference 
skills, and were all items that had 
been assessed in both 1970-71 and 
1974-75. The results for this por- 
tion of the 1979-80 assessment are 
reported in Three National Assess- 
ments of Reading: Changes in Perform- 
ance, 1970-80 (1981). 

In considering new items, the advi- 
sory committee felt that the 1979-80 
objectives called for a wide range of 
reading materials spanning a contin- 
uum from utilitarian to poetic uses 
of language, requiring many kinds of 
inferencing, covering many topics of 
interest and appearing in many for- 
mats and contexts. Accordingly, one 
of the first steps in exercise devei- 
opnent was to specify and assemble 
such texts. 

toother step was to decide which ob- 
jectives should receive the most at- 
tention at each age. Table 3 shows 
the percentage of the assessment 
items devoted to each objective. 



TABLE 3. Relative Weight of Objectives by Age — 
Figures Used to Guide 1979-80 Exercise Developnent 





Age 9 


Age 13 


Age 17 


Valuing 


10% 


10% 


10% 


Comprehension 


45 


35 


35 


Responding 


JO 


30 


30 


Study skills 


10 


20 


20 


General background information 


5 


5 


5 
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Still another consideration was the 
relevance, after almost a decade, of 
materials from the first literature 
assessment. Consultants with exper- 
tise in literature instruction exam- 
ined the old literature exercises in 
light of the new objectives and the 
extent to which certain exercise 
types proved useful the f"rst time. 
In this process, some exercises were 
winnowed out while others were re- 
tained but not recategorized in terms 
of the new objectives. The rest 
(open-ended exercises) were recatego- 
rized in terms of the new objectives. 

Also, prior to full-scale development 
of assessment materials, specifica- 
tions were created for the entire 
assessment. The specifications pro- 
vided overall direction by estimating 
how many exercises of particular 
types are necessary to adequately 
measure achievement of a particular 
objective cr subobjective. They also 
provided detailed directions to exer- 
cise writers, specifying text type, 
measurement goals, domains being as- 
sessed and procedures for determining 
key aspects of text suitable for 
testing comprehension. 

After the objectives had been roughed 
out, it was hoped that a process of 
"objectives amplification 1 * would sup- 
ply the specifications as well as a 
range of model exercises that exer- 
cise writers would only need to imi- 
tate. A series of meetings were de- 
voted to amplification — exhaus- 
tively defining the subparts of any 
objective, modeling the reader- text 
interactions that would presumably 
take place, describing the behaviors 
that would indicate successful 
achievement, establishing measurement 
parameters for each behavior and 
choosing suitable text types. The 
meetings generated thoughtful papers 
and considerable evidence that read- 
ing is a complicated process, but 



they did not generate the direct link 
between objectives and exercises that 
some advisors had hoped for. That 
link remains, in reading and litera- 
ture as in other areas, indirect. But 
the many papers and discussions about 
the leading ptuueSt» ana now to assess 
it did establish a loose, conceptual 
paradigm useful for guiding exercise 
development and interpreting results 
later on. 

The paradigm can be roughly called a 
psycholinguistic model of the reading 
process because it assumes that mean- 
ing resides both in the reader and in 
the text and that comprehension in- 
volves both "top down" and "bottom 
up" kinds of cognitive processing. 
These assumptions affect (1) what 
k i nd of text can be used ( natural ly 
occurring text) , (2) which parts of a 
text an exerc i se wr i ter can use to 
assess a particular level of compre- 
hension, (3) what kinds of distrac- 
tors might be used and (4) how one 
might interpret the results of any 
question about a text. An assessment 
based even loosely upon psycholin- 
guistic theory differs from one based 
on some other general model of tne 
reading process primarily in its em- 
phasis on comprehension as a holistic 
activity instead of a battery of mea- 
surable subskills. But whether the 
differences go beyond that in practi- 
cal terms remains to be seen. 



Item Development, Field Test, 
Rev i ew and Sol ec t i on 
Procedures 

Item writers worked in small groups 
at two centers to create items, ^nd 
the groups then critiqued eaci, 
others' vrork. Once items had been 
developed, critiqued and revised, 
they were revi ewed by Na t ional 
Assessment staff. The results of 
these reviews w^re compiled and item 




writers once again revised items. 
Surviving items were field-tested in 
schools across the country to dis- 
cover potential problems in wording, 
directions or administrative proce- 
dures and to collect item statistics, 
timing information 2nd scoring infor- 
mation. Schools were selected to 
represent high- and low-income commu- 
nities as well as more typical commu- 
nities. Tryout (field-tested) Items 
were administered to student? in at 
least four classrooms (approximately 
100 students) at each of the ages 
assessed. 



So that the field test would closely 
simulate the actual assessment field 
procedures, students recorded their 
answers in the exercise booklets; 
students heard directions and ques- 
tions from an audio tape; and Na- 
tional Assessment staff members, 
rather than classroom teachers, ad- 
ministered the test. The students' 



responses to the items, as well as 
the administrators' reports of any 
field problems, helped both staff and 
consultants to evaluate and revise 
the exercises, Revised exercises were 
generally field-tested again. 

After exercises were field-tested, 
the results were reviewed by National 
Assessment staff and panels of con- 
tent experts, educators and lay per- 
sons from across the country. Exer- 
cises for each age group were re- 
viewed for appropriateness by 
teachers who teach students at that 
age. Lay citizens, representing a 
variety of occupations and interests, 
also reviewed the exercises, checking 
for sex or racial/ethnic bias and 
considering the general importance of 
each exercise. The advisory committee 
worked with NAEP staff to examine the 
items judged to be successful by the 
review panels, and then the staff 
made the final selection of the items 
included in the 1979-80 assessment. 



CHAPTER 3 
PREPARATION OF ASSESSMENT MATERIALS 



Preparation of Booklets 
and Audio Tapes 

National Assessment uses a matrix 
sampling approach, with different 
nationally representative samples of 
students responding to different item 
booklets (see Chapter 4 for details) • 
Since the Assessment's aim is to de- 
scribe results for groups of students 
(males, blacks, students in the West, 
and so on) , not individuals, it is 
not necessary for each student to 
respond to all the items. Each stu- 
dent responded to one booklet of 
items designed to be completed in a 
single class period. 

Following the selection of exercises 
to be included in the assessment, 
National Assessment staff determined 
trfiich exercises were to be included 
in the various booklets and sequenced 
them within the booklets. Booklets 
were constructed separately for each 
age level since students at different 
ages received different sets of exer- 
cises. Thus, exercises for 9-year- 
olds were not sequenced the same as 
those for 13-year-olds, and so forth. 
In 1979-80, there were 11 exercise 
booklets that contained reading/lit- 
erature exercises for 9-year-olds, 15 
such booklets at age 13 and 14 such 
booklets for 17-year-olds. Three 
booklets at each age contained reas- 
sessed reading items (items that had 
been included in two prior assess- 
ments). The remaining booklets were 
comprised of newly developed items. A 
few reassessed literature items from 



the 1970-71 assessment were dispersed 
across all packages — those includ- 
ing reading change and those includ- 
ing newly developed items. 

The following constraints were ob- 
served in preparing the 1979-80 exer- 
cise booklets: 

1. Each booklet contained exercises 
of varying difficulty so that 
students would not become bored 
by many easy exercises or dis- 
couraged by many difficult exer- 
cises. 

2. Exercises could not cue other 
exercises. In other words, the 
answer to one exercise could not 
be contained in another exercise 
in the same booklet. 

3. Each booklet was timed so that it 
would take no more than 45 min- 
utes — the length of a typical 
class period — of a student" s 
time. Booklets contained approxi- 
mately 30-35 minutes of exercise 
time and an additional 10-15 min- 
utes of introductory material, 
instructions and background ques- 
tions. 

4. Booklets were designed to be, 
insofar as possible, parallel 
with respect to the number of 
different objectives measured and 
difficulty levels. Items measur- 
ing a particular objective were 
scattered throughout the booklets 
so that many different students 
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would respond to questions re- 
lated to a particular objective. 

National Assessment has constantly 
attempted to institute procedures to 
minimize difficulties connected with 

*- — 1 •.«k.««ik{ AP *-l — . *- v»__*«l 

will be, as nearly as possible, an 
accurate reflection of vfa$t students 
know and can do. Considerable effort 
was devoted to developing" c^ar in- 
structions and procedures tfc help 
students perform as well as possible 
in the assessment situation. For ex- 
ample, students marked or wrote their 
responses directly in the assessment 
booklets, not on separate answer 
sheets. It was felt that this proce- 
dure WDuld reduce possibilities for 
confusion in using additional sheets 
of paper, especially for the younger 
students. To minimize guessing, stu- 
dents were encouraged to select the 
H I don f t know" response option in- 
cluded with multiple-choice items. 

Paced audio tapes were prepared for 
each exercise booklet to ensure uni- 
form assessment conditions across the 
country and to move students through 
the booklets at the same speed. Stu- 
dents listened to directions on the 



tape for answering exercises, but 
they had to read the texts *nd the 
response choices themselves* 



Differences in Item 
Booklets in the 1970-71, 1974-75 
and 1979-80 Assessments 

National Assessment attempts to make 
assessmenl conditions for items meas- 
uring change identical from assess- 
ment to assessment so that any 
changes observed will be attributable 
to changes in achievement rather than 
a response to an altered testing con- 
dition. Although items were kept 
identical, the makeup of the item 
booklets was different for each read- 
ing, literature or reading/literature 
assessment. Tn 1970-71, reading and 
literature items were included in the 
same booklets, in 1974-75, reading 
items appeared only with other read- 
ing items; and in 1979-80, reading 
and literature again appeared with 
each other (excepting reading change 
items, which were packaged essen- 
tially the same way in the last two 
assessments) . 
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CHAPTER 4 
SAMPLING 



This chapter gives an overview of the 
procecalres used in designing and 
selecting the National Assessment 
samples for the 1970-71, 1974-75 and 
1979-80 reading, literature and 
read i ng/1 i tera t ure assessments . 
Sample design and selection for these 
assessments were conducted by the 
Research Triangle Institute, Raleigh, 
North Carolina, and monitored by 
National Assessment staff. 

The target populations for each of 
the assessments included 9-year-olds 
and 13-year-olds attending school, as 
well as 17-year-olds, both those 
enrolled in either public or private 
schools at the time of the assessment 
and those who either left school 
before graduating or graduated early. 
Adults, 26-35 years old, were also 
assessed in the 1970-71 assessment. 
However, assessment of adults is 
quite expensive, and in 1974-75 and 



i For detailed information about the 
1979-80 National Assessment sampling 
procedures, see Final Report. . .Sam- 
pi ing and Weighting Activities for 
Asse ssment Year 11 (1981) . TTie Na- 
tion al Assessment Approach to Sam- 
pling (1974) gives detailed informa- 
tion about the 1970-71 assessment, 
^Definition of 1979-80 assessment age 
groups are: 9-year-olds — born dur- 
ing calendar year 1970; 13-year-olds 
— born during calendar year 1966; 
and 17-year-olds — born October 1, 
1962, through September 30, 1963. 



1979-80, funds to assess this popula- 
tion were not available. Age-eligible 
persons who were functionally handi- 
capped to the extent that they could 
not participate in an assessment were 
not considered part of the target 
popilations. Specific groups ex- 
cluded were: non-English-speaking 
persons, those identified as nonread- 
ers, persons physically or mentally 
unable to respond and persons in 
institutions or attending schools 
established for the physically or 
mentally handicapped. 

National Assessment did not follow up 
specific individuals from one assess- 
ment to the next. In other words, the 
students who participated in the 
first assessment were not the same 
oner who participated in the second 
or third assessment. However, in each 
assessment year, participants were 
carefully selected to represent each 
age level. For example, at age 9, 
although different sets of probabil- 
ity samples were used for the three 
assessments, each set contained 
nationally representative samples of 
the population of students who were 
nine years old during that assessment 
year. Thus, if we say that 9-year- 
olds 1 ach i evement improved between 
1975 and 1980, we mean that more stu- 
dents who were nine years old in 1980 
correctly answered the same questions 
than did so in 1975. 

Ttie National Assessment samples were 
designed to provide approximately 
2,500 respondents per exercise. Ttiese 
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nunbers allow reporting of data for 
the nation and for the subq roups de- 
fined in Appendix B. In 1974-75, each 
booklet of exercises was administered 
to approximately 7,500 respondents, 
or triple the usual sample. This pro- 
cedure enabled analysis and reporting 
of 1974-75 results for additional 
subgroups. 

Overview of the National 
Assessment Sample Design 

For all of its assessments, NAEP uses 
a deeply stratified, three-stage na- 
tional probability sample design with 
oversampling of low-income and rural 
areas. In the first stage, the United 
States is divided into geographical 
units. In 1970-71 and 1974-75, these 
units were counties or groups of con- 
tiguous counties meeting a minimum 
population size requirement. In 
1979-80, some counties containing 
large population centers were divided 
into more than one geographical unit. 
These units, called primary sampling 
units (PSUs), are stratified by re- 
gion and size of community- From the 
list of PSUs, a sample of PSUs is 
drawn with probability proportional 
to population size measures, repre- 
senting all regions and sizes of com- 
munities. Oversampling of low-income 
and extreme-rural areas is first per- 
formed at this stage by adjusting the 
estimated population size measures of 



such areas to increase sampl ing 
rates. In the current sampling proce- 
dures, Census Employment Survey Data 
are used within PSUs to further de- 
lineate and oversample , low-income 
areas. Counties with high proportions 
of rural families are also oversam- 
pled. 

In the second stage, all public and 
private schools within each PSU se- 
lected in the first stage are listed. 
Schools within each PSU are selected 
without replacement with probabili- 
ties proportional to the number of 
age-el igibles in the school. 

Ttie third stage of sampling occurs 
during the data collection period. A 
list of all age-eligible students 
within each selected school is made. 
A simple random selection of eligible 
students without replacement is ob- 
tained, and item booklets are admin- 
istered to selected students. Spe- 
cially trained personnel select the 
sample and administer the booklets. 

Survey Weights 

The number of PSUs, schools within 
PSUs and students within schools are 
determined by optimum sampling 
principles. That is, a sample design 
is utilized that will achieve the 
maximum precision for a rjiv^n level 
of resources. Table 4 displays the 



TABLE 4. Number of PSUs Selected and Schools 
Within PSUs Included in the Assessmer w 
in 1970-71, 1974-75 and 1979-80 



Age 9 
Age 13 
Aqe 17 



1970-71 
Assessment 



1974-75 
Assessment 



1979-80 
Assessment 



No. of 


No. of 


Mc. of 


No. of 


No. of 


No. of 


PSUs 


Schools 


PSUs 


Schools 


PSUs 


Schools 


116 


1,007 


115 


1,003 


83 


560 


116 


1,029 


115 


972 


83 


534 


116 


631 


115 


830 


83 


412 
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number of PS Us used and the nunber of 
schools in which assessment sessions 
were conducted, by age, for the 
1970-71, 1974-75 and 1979-80 
assessments. Appendix E gives 
information about the nunber of 
students assessed. 

In order to locate those 17-year-olds 
who had left school prior to the 
assessment, half the schools in each 
PSU were randomly selected and asked 
to provide lists of potentially 
eligible dropouts for the three ost 
recent school years and, for schools 
having 12th grades, lists of 
potentially eligible early graduates. 

After receipt from the schools, the 
dropout and early graduate lists were 
screened to eliminate persons with 
ineligible birth dates and duplicate 
listings, and to establish the 
final-stage sampling frame of 
potentially eligible individuals for 
each school. The field staff 
attempted to locate each of the 
individuals in the sample and assess 
those found to be eligible. 
Respondents were given the 
opportunity to complete up to four of 
the assessment booklets and were 
remunerated at the rate of $5.00 for 
each completed booklet. 

« 

The 1970-71 assessment included an 



additional household survey aimed at 
both 17-year-olds not enrolled in 
school and young adults 26 to 35 
years of age. However, this method of 
sampling is extremely costly and 
1 imi ted resources prohibi ted 
including such sampling techniques in 
the 1974-75 and 1979-80 assessments. 
Thus, for those two assessments, 
young adults were not included and 
17-year-olds were located only by 
using the school sample design. Since 
only a small percentage of 
17-year-olds not enrolled in school 
were found by using the household 
survey method, differences in the 
assessment samples are minimal. 

Each respondent in the sample does 
not have the same probability of 
selection because sane subpopjlations 
are oversampled and because 
adjustments are made to compensate 
for student nonresponse and for some 
schools' refusals to participate. The 
sel ect ion probabi 1 i ty for each 
individual is computed, and its 
reciprocal is us*d to weight each 
response in any statistical 
calculation to compensate for unequal 
rates of sampling and to ensure 
proper representation in the 
population structure. Procedures 
used to assign weights are discussed 
in Chapter 7 and Appendix F. 
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CHAPTER 5 
DATA COLLECTION 



National Assessment subcontracted 
data collection to the Research Tri- 
angle Institute for the 1979-80 read- 
ing/literature assessment. A profes- 
sional data collection staff was used 
rather than school personnel to mini- 
mize the burden on participating 
scnools and to ensure, insofar as 
possible, uniform administrative con- 
ditions across the country ( Final 
Report...ln-School Field Opera- 
tions. .. , 1980 ). In all three assess- 
ments, NAEP staff worked closely with 
the subcontractors to enisure adher- 
ence to rigorous administrative stan- 
dards. 

Participation in the National Assess- 
ment is voluntary* NAEP makes every 
effort to encourage the schools se- 
lected in the sample to participate 
in the assessment, and National 
Assessment and Research Triangle In- 
stitute staffs have obtained high 
rates of school cooperation, as shown 
in Table 5 uinal Report. ».In~School 
Field Operations. .. , 1980, p. 40, 



TABLE 5. School Cooperation 
Rates, 1979-80 Assessment 

Age Percent of Eligible Schools 
Participating in 1979-80 
Assessment 

9 94.5 

13 93.2 

17 90.5 

Overall 92.9 



Table 28) . Student cooperation rates 
were also high. 

Each age group was assessed at ap- 
prox ima tely the same t ime of the 
school year in each of the assess- 
ments: 13-year-olds were assessed in 
October-December, 9-year-olds in Jan- 
uary-February, in-school 17-year-olds 
in March-May and out-of-school 17- 
year-olds in June-July. In 1979-80, 
booklets were administered to groups 
of 10-25 students, with each group 
responding to only one of the book- 
lets for their age level, except for 
out-of-school 17-year-olds who took 
up to three parities each. The groups 
varied in size depending on the num- 
ber of eligible students_ J artd an esti- 
mate of the rate of nonresponse for a 
particular school. In 1970-71 and 
1974-75, the planned session sizes 
were fixed at 12 students. 

In each assessment, steps were taken 
to guarantee the anonymity of respon- 
dents. Students* names were listed 
with air booklet identification 
nur.ber so that scoring and process inq 
personnel could go back to the school 
lists for data verification — for 
instance, on background information 
— if necessary. These lists did not 
leave the schools and were destroyed 
six months following the assessment 
in a school. 

Tb provide information on respondents 1 
backgrounds, school officials were 
asked to respond to a n principal's 
quest ionnai re, * which included ques- 
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tions about the size and type of com- 
munity served by the schools. In ad- 
dition, in 1979-80, officials in 
schools were asked to respond to a 
"supplementary principal's question- 
naire," which asked about read- 
ing/li teratar e programs in the 
school. Students also provided in- 
formation on their backgrounds 
through questions included in the 
item booklets. Samples of forms used 
to collect background information 
from students and school officials in 
the 1979-80 assessment appear in Ap- 
pendix C. 

The assessment administrator coded 
each student's birth date, sex, 
grade, racial/ethnic classification 
and identification number on his or 
her booklet. Administrators made a 
visual racial/ethnic identification 
at the time each student turned in 
his or her booklet. During the 1979- 
80 assessment, six different racial 
classifications were usedr white, 
black, Spanish heritage, American 
Indian or Alaskan native, Pacific 
Islander or Asian, and unclassif ied* 
If an administrator was unsure of a 
student's racial/ethnic group, the 
administrator referred to the stu- 
dent's name or listened to the stu- 
dent talk to make the identification. 



The assessment administrators did not 
ask students to give a racial identi- 
fication for themselves; however, in 
1979-80 17-year-old students were 
asked to provide this information in 
one of the background questions in- 
cluded in each exercise booklet. 

Sample sizes of the classifications 
American Indian or Alaskan native and 
Pacific Islander or Asian were too 
small to permit reporting for these 
groups. Also, results for the group 
classified as Spanish heritage cannot 
be reported for separate exercises, 
only for aggregates of exercises. 



Following data collection, assessment 
administrators sent comfieted book- 
lets to the scoring contractor, West- 
inghouse DataScore Systems, Iowa 
City, Iowa. Booklets were checked to 
verify that correct administrative 
procedures were followed by the field 
staff. Coded identification informa- 
tion was also checked for accuracy; 
inconsistencies that ecltild not be 
reconciled were sent back to the 
assessment administrator to be 
checked against the list of student 
names and identification numbers re- 
tained by the school for six months 
following the assessment. 
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CHAPTER 6 
SCORING 



Scoring and computer record irxj of 
data were contracted to Westing ho use 
DataScore Systems, 1 Iowa City, Iowa, 
for all three reading assessments. 
National Assessment has found it most 
efficient to have scoring done by an 
outside contractor and to have the 
same contractor do both the machine 
scoring and the open-ended, or hand f 
scoring. Booklets to be scored do not 
have to be shipped to another loca- 
tion when different scoring method^ 
are needed; in addition, the scoring 
contractor has a trained staff of 
scoring personnel that can be called 
upon and augmented when National 
Assessment conducts a major scoring 
effort. 

In the 1979-80 assessment, more than 
90% of the items were multiple-choice 
and the rest were open-ended. Re- 
sponses to multiple-choice exercises 
were read directly by an optical 
scanning machine. The scoring con- 
tractor employ a special staff to 
hand score open-ended exercises. 
Scorers were responsible for catego- 
rizing open-ended responses, using 
scoring guides thac defined cate- 
gories of acceptable and unacceptable 
responses. They then coded this in- 
formation into ovals that could be 
read by the optical scanning machine. 

For changes in performance to be 



Formerly the Measurement Research 
Center, Iowa City, Iowa. 



measured , lately, scoring had to 
be the same for responses collected 
in each assessment year. For multi- 
ple-choice items, the same responses 
were scored correct in each ye^r. 
Some open-ended items were short- 
answer reading items requiring rather 
objective scoring of a clerical na- 
ture. These were all reassessed 
Hems, and identical guides were used 
in 1979-80 as were used to categorize 
the 1970-71 and 1974-75 responses. 
Scorers were trained using sample 
responses from all three assessment 
years. Qiality-control procedures 
were conducted by having scorers re- 
score papers from previous assess- 
ments along with the 1979-80 respons- 
es. A 5% subsample from each previous 
assessment was rescored, and percent- 
ages of agreement w*th the earlier 
scorings averaged approximately 99%. 

Most of the open-ended scoring effort 
was concentrated on a variety d¥ ex- 
ercises that required at least para- 
qrapn length open-ended responses to 
poems and prose passages, Tfost of 
these items were developed for the 
1979-80 assessment , al though a few 
were 1970-71 literature items read- 
ministered to measure changes in per- 
formance. Scoring guides 4 for these 
newly developed items were con- 
structed using both field tryout data 
and actual assessment data. Scoring 
guides for the few reassessed items 
were revised in 1979-80, using both 
1970-71 and 1979-80 assessment data, 
to be consistent with the guides con- 
structed for items first administered 
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in 1979-80. To ensure that scoring of 
the two sets of assessment data was 
identical, all 1970-71 responses to 
open-ended literature items were re- 
scored at the same time that the 
1979-80 responses were scored. 

Although the use of a variety of 
types of tasks and scoring guides 
increases the expense and complexity 
of the open-ended scoring task, it 
nevertheless provides a more compre- 
hensive means of assessing students 1 
abilities to respond to written 
works. Five different types of open- 
ended exercises were included in the 
1979-80 assessment of "response to 
written works," Each required differ- 
ent skills and levels of ability on 
the pert of the respondents and as 
such, required a different scoring 
procedure. Each type of responding 
task and scoring procedure is dis- 
cussed below. * 

Gener al respo nding tasks asked re- 
spondents to discuss the passage or 
poem presented, or to describe their 
thoughts or feelings about the text. 
Tt was expected that responses to 
this type of item would be highly 
text-dependent and would allow the 
writer to select from a variety of 
perspectives. Ttie writer was given 
very little explicit focus for his/ 
her response, and was therefore free 
to choose whichever approach seemed 
most natural. The scoring guide cate- 
gories for this type of exercise are 
descriptive and do not readily iSnd 
themselves to quantitative ranking. 
The response categories scored were: 
egocentric, personal, emotional, re-, 
telling, inferencing, generalization, 
analysis, reference to other wrks 
and evaluation. At age 17 only, three 
of these categories were further di- 
vided into two levels each to provide 
more qualitative information; analy- 
sis (le^/el 1 = superficial, level 2 = 
elaborated), oi"her wrks (level 1 = 
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general, level 2 = specific) and per- 
sonal (level 1 = global, level 2 - 
analytic) . 

h second type of open-ended exejeise 
is referred to as inferenc inc;, and 
infer enc i ng ta sks required either 
general or specific inferences relat- 
ing to the mood of, or a character 
in, the passage. Respondents were 
asked to describe the intent of the 
author, to describe the mood of the 
passage or to describe the character 
of the protagonist. These kinds of 
items require the reader to interpret 
the passage and to explain the inter- 
pretation by relating it to the text. 

Qualitative differences in score 
points^ were dependent upon the iden- 
tification of the intent, mood or 
character traits and upon the amount 
and nature of the supporting evidence 
provided. In addition, descriptive 
data were obtained concerning the 
source of- the evidence — whether it 
related to the content or form of the 
text, or whether it represented a 
subjective reacrion on the part of 
the reader — and the number of 
pieces of evidence provided by the 
writer. Qualitative ranks were as- 
signed and, again, additional de- 
scriptive information was obtained. 

Emotional resp onding tasks asked re- 
spondents to describe emotions or 
feelings aroused by the text. Itie 
scoring guide categorized both the 
identification of the emotion (or 
feeling) and the presentation of evi- 
dence supporting the emotion. 

Analytic responding tasks as*ked the 
respondents to analyze a passage or 
poem. Successful responses were those 
that went beyond a superficial inter- 
pretation and provided a theme or 
meaning for the text. In add;* inn, it 
was necessary that the respondents 
discuss the way in which some 



feature(s) of the text contributes to 
the statement of the thane. These 
responses received only qualitative- 
rank scores. N-> further descriptive 
information was obtained. 

The final type of open-ended 
exercise, evaluative responding 
tasks , asked students to evaluate 
particular poems or stories. Scoring 
guides for these exercises measured 
the respondents' abilities to state 
their criteria, and where 
appropriate, to provide examples from 
the text that were related to the 
criteria. CXialitative ranks were 
assigned to the war ious response 
types, and descriptive information 
was also assessed. I 

Westinghouse DataScc^re Systems and 
National Assessment /staff worked to- 
gether to train readers. In training 
sessions, readers were given the 
scoring guide for an item and re- 
sponses that exemplified each scoring 
category. The reasons why responses 
were classified in particular cate- 
gories were discussed; sce.ers' ques- 
tions were answered and, if neces- 
sary, modifications were made to 
scoring guides. Readers then scored 
several papers and categorizations 
were discussed. This process contin- 
ued until readers were familiar with 
the application of the scoring guides 
and was repeats for each task and 
separate age group assessed to be 



sure that scoring was consistent. 

Tb further ensure the quality and 
consistency of scoring open-ended 
exerc i ses , qual i ty-con trol checks 
were conducted during the scoring of 
these exercises. At regular inter- 
vals, randomly selected responses 
were - drawn from the total pool of 
responses for an item and read by 
randomly selected scorers. Both the 
responses and the scorers were se- 
lected without replacement; approxi- 
mately 10% of the responses were in- 
cluded in the quality-control check. 
Scores fcr the quality-control read- 
ers were recorded, and the responses 
selected for quality control were 
then put back into the total pool <3f 
responses to be scored during the 
regular course of scoring. Following 
scoring of all responses, the two 
scores for quality-control responses 
were compared. If discrepancies in 
scoring became apparent, scorers were 
retrained and, on some occasions, 
work was rescored. 

Percentages of agreement between 
quality-control and regular scoring 
were computed for each open-ended 
e::ercise. These data are summarized 
in Table 6. 

Scoring for each age group began dur- 
ing the administration of the assess- 
ment to that age group. 
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TABLE 6. Average Percentages of Scorers' 
Agreement for 1979-80 Open-Ended Scoring, 
by Age Croups 



Title Average % of Agreement 





Age 9 


Age 13 


Age 17 


Somebody's Son 




95.0 


95.0 


Somebody's Son (essay) 




95.0 


93.0 


Good Dog 





95.0 


94.0 


One of These Days 






94.0 


Old Dog 


98.0 


91.0 




Mother to Son 




91.0 


91.0 


i was you 






93.0 


Good Story 


98.0 


93.0 


94.0 


Good Poem 


98.0 


96.0 


94.0 


Check 




96.5 




Into My Heart 






96.5 


As the Cat 


96.5 






Rodeo 




94.0 


94.0 
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CHAPTER 7 
DATA ANALYSIS 



Measures of Achievement 

Hie basic measure of achievement re- 
ported by National Assessment is the 
percentage of respondents answering a 
given item acceptably. This percent- 
age is an estimate of the percentage 
of 9-, 13- or 17-year-olds who would 
respond acceptably to a given item if 
every 9-, 13- or 17-year-old in the 
country were assessed. 

Percentages of acceptable responses 
are used because each item is 
designed as a separate measure of 
some aspect of an objective or 
subobjective. Hie purpose of National 
Assess* lent is to discover if more or 
fewer people are able to answer these 
items acceptably — and thus meet the 
objectives — over time. 

In addition to providing results on 
individual items, National Assessment 
reports the average performance 
across groups of similar items — for 
the learning area as a whole, for a 
particular theme, objective or sebob- 
ject i ve , and so on . These resul ts 
constitute the mean or arithmetic 
average of the estimates of perform- 
ance on the group of items and is 
called the mean percentage correct. 



TWenty-two empirical distributions 
of change measures from the 1969-70 
and 1972-73 science assessments were 
used to generate tonte Carlo simula- 
tions of sampling distributions for 



The items included in the calculation 
of a mean percentage usually are lo- 
cated in several exercise booklets 
and, thus, the mean percentage should 
not be construed as an average test 
score. 

To present a general picture of 
changes in achievement, National 
Assessment describes the gains or 
losses on a group of exercises in 
terms of the differences in the aver- 
age percentage of acceptable respons- 
es. 

Unless the items summarized in the 
mean percentages of acceptable re- 
sponses are identical, the means of 
one age group should not be compared 
with the means of another, since 
their values reflect both the choice 
of exercises and the performance of 



several measures of central location. 
In addition to the mean and median, 
other measures of central location 
that were considered in the simula- 
tion studies included the average of 
the extremes, two forms of bi weigh ted 
estimates and three forms of weight- 
matching estimation described by John 
W. TUkey in the research paper "Some 
Considerations on Locators Apt for 
Some Squeezed-Tail (and Stretched- 
Tail) Parents* (1975). In almost 
every case, the sampling stability of 
the mean change was as good as or 
better than that of the other meas- 
ures studied. 
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the students. When only a few exer- 
cises are summarized by a mean, one 
should be especially cautious in in- 
terpreting results, since a small set 
of exercises might not adequately 
cover the wide range of potential 
behaviors included under a given ob- 
jective or subobjective. The mean 
should be interpreted literally as 
the arithmetic average of the per- 
centage of acceptable responses ob- 
tained from National Assessment sam- 
ples on a specific set of exercises. 
It should not be construed as an av- 
erage test score . 

In addition to providing national 
results, National Assessment reports 
on the achievement of various subpop- 
ulations of interest. Groups are 
defined by region of the country, 
sex, race/ethnicity, size and type of 
community lived in, level of parents* 
education and grade in school (see 
Appendix B for definitions of these 
groups) . 

The difference between percentages or 
averages for a reporting group and 
that of the entire age group (nation) 
on an exercise is used to describe 
the performance of any reporting 
group relative to the entire age 
group. Ttlis difference is a positive 
number if the group achieved a higher 
percentage or average than the entire 
age group and is a negative, number if 
the group achieved a lower percentage 
or average. For exanple, a group per- 
formance of +1.8% indicates that the 
percentage of responses for the group 
is 1.8 percentage points higher than 
the national percentage of responses 
for that age level. 



in considering National Assessment's 
achievement measures, it is differ- 
ences in performances between assess- 
ments, among groups and among ages, 
that are the most useful. By main- 
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taining the same item or set of items 
in making these comparisons, we have 
a reasonable indicator of whether 
more or fewer people know or can do 
something judged important. 

Procedures for estimating percentages 
of responses to exercises are 
dependent on the sample design. Each 
response by an individual was 
weighted and mul t i pi i ed by an 
adjustment factor for nonresponse. 2 
An estimate of the percentages of a 
particular age group that would have 
responded to a particular exercise in 
a particular way if the entire age 
group were assessed was defined as 
the weighted number of that type of 
response divided by the weighted 
number of all the responses. A 
similar ratio of weights was used to 
estimate percentages and averages for 
reporting groups or subpopulations of 
interest. 



Estimating Variability in 
Achievement Measures 

National Assessment used a national 
probability sample at each age level 
to estimate the proportion of people 
who would complete an exercise in a 
particular way. The sample selected 
was one of a large number of all pos- 
sible samples of the same size that 
could have been selected using the 
same sample design. Since an achieve- 
ment Measure computed from each of 



Appendix D discusses nonresponse in 
assessment samples. 

3 Following the 1979-80 assessment, a 
weighting^class adjustment procedure 
was used to anooth estimated popula- 
tion proportions across the 10 as- 
sessments conducted between 1970-71 
and 1979-80. A discussion of this 
procedure is included in Appendix F. 
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the possible samples would differ 
from one sample to another, the stan- 
dard error of this statistic was used 
as a measure of the sampling varia- 
bility among achievement measures 
from all possible samples. A standard 
error, based on one particular sam- 
ple, serves to estimate that samplinq 
variability. 

In the interesc of sampling and cost 
efficiencies, National Assessment 
uses a complex, stratified, multi- 
stage probability sample design. Typ- 
ically, complex designs do not pro- 
vide for unbiased or simple computa- 
tion of sampling errors. A reasonably 
good approximation of standard error 
estimates of acceptable response per- 
centages and averages wap obtained by 
applying the jackknil procedure 
(Miller, 1964, pp. 1594-i J5; Miller, 
1968, pp. 567-82; Mosteller and 
Tukey, 1968) to first-stage sampling 
units within strata. Standard errors 
for achievement measures such as na- 
tional percentages, group differ- 
ences, means or mean differences for 
a particular assessment year were 
estimated directly, taking advantage 
of features of the jackknife proce- 
dure that are generic to all of these 
statistics. Since samples for dif- 
ferent assessments are independent, 
the standard errors of the differ- 
ences in achievement measures between 
assessments can be estimated .simply 
by the square root of the sun of 
squared standard errors from each of 
the assessments. 

Hie standard error provides an esti- 
mate of sampling reliability for the 
achievement measures used by National 
Assessment. It is comprised of sanv- 



See Appendix E for a more detailed 
description of National Assessment's 
compjtation of standard errors. 



pling error and other random error 
associated with the assessment ot a 
specific item or set of items. Random 
error includes all possible nonsys- 
tanatic error associated with admin- 
istering specific exercises to spe- 
cific students in specific situa- 
tions. Random differences among scor- 
ers for open-ended^ items are also 
included in the standafd^rrors. 

National Assessment has adhered to a 
standard convention whereby differ- 
ences between statistics are desig- 
nated as statistically significant at 
the .05 level of significance. That 
is, differences in performance be- 
tween assessment years or between a 
reporting group and the nation are 
highlighted with asterisks only if 
they are at least twice as large as 
their standard error. Differences 
this large would occur by chance in 
fewer than 5% of all possible repli- 
cations of our sampling and data col- 
lection procedures for any particular 
reporting group or national esti- 
mates. 



Controlling Nonrandom 
Errors 

Systematic errors can be introduced 
at any stage of an assessment — ex- 
ercise development, preparation of 
exercise booklets, design or adminis- 
trative procedures, field administra- 
tion, scoring or analysis, These 
nonsampling, nonrandom errors rarely 
can be quantified, nfcr can the magni- 
tude of the bias they introduce into 
the estimates be evaluated directly. 

Systematic errors can be controlled 
in large part by employing uniform 
administrative and scoring procedures 
and requiring rigorous quality con- 
trol in all phases of an assessment. 
Tf the systematic errors are the same 
from age to age or group to group, 
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then the differences in percentages 
or mean percentages are measures with 
reduced bias because subtraction 
tends to cancel the effect of the 
systematic errors. 

Similarly, the effect of systematic 
errors in different assessment years 
can be controlled by carefully repli- 
cating in the second assessment the 
procedures carried out in the f irst. 
Differences in achievement across 
assessment years will also be meas- 
ures with reduced bias since subtrac- 



tion will again tend to cancel sys- 
tematic errors. 



Although it is not possible for every 
condition or procedure to remain ex- 
actly the same between assessments 
conducted several years apart, Na- 
tional Assessment has made every ef- 
fort to keep conditions as nearly the 
same as possible. Changes in proce- 
dures described in this report were 
judged to have a relatively minor 
impact. 
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REPORTS ABOUT THE 
READING AND LITERATURE ASSESSMENTS 



Each assessment generates a tremen- 
dous amount of information. To make 
the data as useful as possible to a 
variety of audiences, National 
Assessment provides several types of 
publications. 

Reports 

Since it is difficult and time- 
consuming to synthesize many discrete 
bits of data, National Assessment 
prepares reports about the reading 
and literature assessments for the 
general public — including parents, 
classroom teachers, school adminis- 
trators and legislators — that not 
only provide considerable data about 
each exercise but also synthesize and 
highlight assessment results. 

Although National Assessment does not 
interpret assessment results, it rec- 
ognizes that data presented alone are 
often difficult to consider in per- 
spective. Accordingly, National 
Assessment asked a group of reading 
educators to review and comment upon 
the results; their comments are in- 
cluded in the reports summarizing the 
reading assessment results. 

A report, Three National Assessments 
of Reading; Changes in Performance, 
1970*80 (1981), concerning changes in 
reading performance across the three 
assessments is available. 

Additional reports on responding to 
written works and performance on com- 



prehension and study skills items by 
9-, 13- and 17-year-old students are 
planned for publication during 1981- 
82. 

The following reports describing re- 
sults from the second assessment are 
also available: 

Reading in America: A Perspective 
on TVro Assessments (1976) . 

Reading Change, 1970-75: Summary 
Volune (1978). 



Objectives 

A description of the 1974-75 reading 
objectives and 1979-80 reading/lit- 
erature objectives and the procedures 
used in developing the objectives are 
available in Reading Objectives, Sec- 
ond Assessment (1974) and Reading and 
Literature Objectives, 1979-8 0 
Assessment (1980) • 



Exercises, Scoring 
Procedures and Data 

For those wishing to use specific 
National Assessment items, NAEP pro- 
vides copies of released items, exer- 
cise documentation and scoring guides 
as well as information on scoring 
procedures and exercise-level data on 
microfiche. Materials that are or 
will be available include: 
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Reading/Literature Released Exer- 
cis e Set, 1979-80 Asse ssment 

Reading/Literature Released Exer- 
cise Set, 1979-80 Assessment ; 
Supplement (1981). 

For those desiring additional exer- 
cises, the exercises released fol levy- 
ing the first assessment are availa- 
ble in Literature; Released Exercises 
(1973) and The First Assessment of 
Reading, 1970-71 Assessment, Released 
Exercise Set (1979). 



Public-Use Data Tapes 

For those who wish to perform their 
own analyses of National Assessment 
data, National Assessment will make 



available data tapes of respondent- 
level data for all reading and 
literature and reading/literature 
assessments. To protect the confi- 
dentiality of the respondents, all 
identifying information (school, dis- 
trict, state) has been deleted. The 
tapes include docurtentation of exer- 
cises and are organized and docu- 
mented in si'ch a way that they can be 
used with standard s .atistical pack- 
ages. 



User Services 

National Assessment provides some 
assistance to those wishing to use 
assessment items or to replicate 
assessment methodology. Those 
interested in receiving assistance 
should contact the National Assess- 
ment office. 
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APPENDIX B 

DEFINITIONS OP NATIONAL ASSESSMENT REPORTING GROUPS 



In addition to reporting results for 
all 9-, 13- and 17-year-old students 
in the United States, National 
Assessment reports results for a num- 
ber of population subgroups. Most of 
these subgroups are defined for both 
the reading and literature assess- 
ments. 

Definitions of the subgroups follow: 



Region 

The country has been divided into 
four regions: Northeast, Southeast, 
Central and West. States included in 
each region are shown on the follow- 
ing map. 




Sex 

Results are reported for males and 
females. 



Race/Ethnicity 

Results are presented for blacks, 
whites and Hispanos (1979-80 only). 



Level of Parental Education 

National Assessment defines three 
categories of parental-education lev- 
els, based on students 1 reports. 
These catego r i es a re : ( 1 ) those 
whose parents did not graduate from 
high school f (2) those who have at 
least one parent who graduated from 
high school and (3) those who have at 
least one parent who has had some 
post-high-school education. 



Type of Conmunity 

Three extreme community types of spe- 
cial interest are defined by an occu- 
pational profile of the area served 
by a school as well as by the size of 
the community in which the school is 
located. 

Advantaged-urban (high-metro) commu- 
nities . Students in this group attend 
schools in or around cities having a 
population greater than 200,000 where 
a high proportion of the residents 
are in professional or managerial 
positions. 

Disadyantaqed-urban (low-metro) com- 
munities . Students in this group at- 
tend schools in or around cities hav- 
ing a population greater than 200,000 
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where a relatively high proportion of 
the residents are on welfare or are 
not regularly employed • 

Rural communities . Students in this 
group attend schools in areas with a 
population under 10,000 where many of 
the residents are farmers or farm 
workers. 

This is the only reporting category 
that excludes a large number of re- 
spondents. About * two- thirds do not 
fall into the classifications listed 
above. Results for the remaining two- 
thirds are not reported since their 
performance was similar to that of 
the nation. 



Size of Community 

Big cities . Students in this group 
attend schools within the city limits 
of cities having a 1970 census popu-* 
la t ion over 200,000. 

Fringes around big cities . Students 
in this group attend schools within 
metropolitan areas (1970 U.S. Bureau 
of the Census urbanized areas) served 
by cities having a population greater 
than 200,000 but outside the city 
limits. 

Median cities . Students in this group 
attend schools in cities having a 
population between 25,000 and 
200,000, not classified in the 
fringes-around-big-cities categorv. 

Small places . Students in this group 
attend schools in communities having 
a populat ion 1 ess than 25 , 000 , not 
classified in the fringes-around- 
big-cities category. 



Grade in School 

Results are categorized for 9-year- 



olds in the 3rd or 4th grade; 13- 
year-olds in the 7th or 8th grade; 
and 17-year-olds in the 10th, 11th or 
12th grade. 



Modal Grade by Region 

Results are categorized for 9-, 13- 
and 17-year-old respondents in grades 
4, 8 and 11, respectively, who live 
in the Northeastern, Southeastern, 
Central or Western region of the 
country. 

Modal Grade by Cmunity Size 

Results are categorized for 9-, 13- 
and 17-year-old respondents in grades 
4, 8 and 11, respectively, who live 
in big cities, fringes around big 
cities, medium cities - and small 
places. 



Modal Grade by Sex 

Results are categorized for 9-, 13- 
and 17-year-old males and females in 
grades 4, 8 and 11, respectively. 



Reading Enjoyment 

Results are categorized for 9-, 13- 
and 17-year-old respondents who re- 
sponded to a guest ion concerning how 
much they enjoy reading. Possible 
responses are: do not enjoy it at 
all, enjoy it somewhat, enjoy it very 
much* 



Kindergarten Attendance 

Results are categorized for 9-year* 
olds only who responded either yes or 
no to a background question concern- 
ing kindergarten attendance. 
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Kind of Reader 



Spare-Time Reading 



Results are categorized for 9-, 13- 
and 17-year-old respondents by poor, 
good and very good readers. The cate- 
gories were created from responses to 
a question asking the students 
whether they thought of themselves as 
poor, good or very good. 



Results are categorized for 9-, 13- 
and 17-year-old respondents specify- 
ing either a preference for reading 
fiction or nonfiction, no preference 
or else stating that they never read 
during their spare time. 



Reading Importance 

Results are categorized 1 for 9-, 13- 
and 17-year-old respondents by type 
of response to a background item re- 
lating to whether reading is very 
important, somewhat important or not 
at all important. 



Reading Frequency 

Results are categorized for 9-, 13- 
and 17-year-old respondents according 
to frequency* of their reading for 
enjoyment. Possible responses are: 
almost daily, once or twice a week, 
less than once a week and never. 

TV Viewing 

Results are categorized for 9-, 13- 
and 17-year-old respondents by amount 
of time spent matching TV tte devi- 
ous day. Possible response:, are: 1 
hour or less, 1 to 2 hours, 3 to 4 
hours, 5 or more hours. 

Language 

Results are categorized for 9-, 13- 
and 17-year-old respondents according 
to language use in the home. Possible 
responses are: English spoken most 
and other language seldom or never, 
English spoken most and other lan- 
guage often, English not most often 
spoken but other language uz^ most 
often. 



Time Spent Reading 

Results are categorized for 9-, 13- 
and 17-year-old respondents by time 
spent reading for enjoyment the pre- 
vious day, reported by* time amounts 
of less than 1 hour, 1 to 2 hours, 3 
or more houl^s. 



m^WOTk-/^ 



Horn* 



Results are categorized for 13- and 
17-year-olds according to responses 
concerning amount of time spent on 
v omework the previous day. Possible 
respDnses are: none assigned, did not 
do any assigned homework, less than 1 
% hour f between 1 to 2 hours, more than 
2 hours. 



Achievement class 

Results are presented in four ranges 
of achievement performance. 

Achievement class 1 . The lowest one- 
fourth of the national sample. 

Achievement class 2 . The middle low- 
est one-fourth of Lhe national sam- 
ple. 

Achievement class 3 . Ttie middle high- 
est one-fourth of the national sam- 
ple. 

Achievement class 4. The highest one- 
fourth of the national sample. 
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APPENDIX C 

FORMS USED TO OBTAIN BACKGROUND INFORMATION 

This appendix contains the forms used 1979-80 assessment* following is a 

by National Assessment to collect list and a brief description of the 

background information from school forms included, 
officials and respondents for the 

p. 37 School Principals Questionnaire — filled out by school 
principals or other school officials for schools at each 
of the age levels discussed. 

p. 39 Supplementary Principal's Questionnaire — f ill6d out by 
school officials to provide information about regular 
school reading/literature programs, remedial and enrich- 
ment courses, teacher support and school resources. 

P* 50 Package Cover Sheet — cover of item booklet filled out 
by exercise administrators to provide information about 
the grade, sex, birth date and race of each student. 

P* 51 Directions to Exercise Administrators for Coding Package 
Cover Sheet — tells exercise administrators how to code 
information in boxes 3-8 on package cover sheet. Direc- 
tions shown are for 13-year-olds* booklets* 

p. 52 Standard Background Information Fom for 9-Year-Olds — 
provides information about reading material in the home 
and level of parents 1 education. 

p. 53 Standard Background Information Form for 13-Year-Olds — 
provides information aoout reading material in the home, 
level of parents 1 education and place lived in at age 9. 

p* 54 Standard Background Information Form for 17-Year-01ds — 
provides information on level of parents 1 education and 
place lived in at ages 9 and 13. 

P* 55 Supplementary Background Information Form for 9-Year- 
Olds — provides information about TV watching, time 
spent reading, kindergarten attendance, English language 
usage and self perception as a reader. 
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57 Supplementary Background Information Form for 13-Year- 
Olds — provides information about homework, TV watch- 
ing, time spent reading, Ehglish language usage and 
self- percept ion as a reader. 

59 Supplementary Background Information Form for 17-Year- 
01 ds — provides information about racial and ethnic 
identification, possessions in the home and primary em- 
phasis of high school courses, in addition to questions 
asked of 13-year-olds. 

62 Reading Background Questionnaire for 17-Year-01ds Out of 
School — provides information about current employment 
and frequency ot reading tasks in job situations. 
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School Principal's Questionnaire 



Thtt rtport is author i/ao* by !a* (20 USC 1221 c-ll WMa 
you art not raoutrad 10 rwpood. your cooparatioo if naadad 
to maka tha rasuftt of thtt survey comprenansiva, accurate, 
and ttmdy 



Primary Sampling Unit [ J] ] 



PLEASE 
PRINT 



School Number [ T | | 



Age Croup(s) 



Name of School 
Address of School 



(Street) 



(City) 

Name of School Principal 



(State) 



13 



17 



(Zip Code) 



Name and title of person completing the form if other than school principal 
Name Title 



What is your best e s tima t e of the current enro llment and the average daily 
attendance by grade of your school (1979-80 school year)? (Enter zeros for 
grades not served by your school.) 



Grade 

Enrollment 

Average 
Daily 

Attendance 



K 


1 


2 


3 


4 


5 


r 

6 J 


7 


8 


9 


10 


11 


1 u 






— — 


■ ■ 




1 


■ 












1 



















— 






, 






Approximately what percentage of the students attending your school live in each 
of the following areas? 

% A In a rural area (less than 2*500) 

1 B In a town of 2,500 to 10,000 
% C In a town of 10,000 or more 



100% 



(Items A-C should add to 100%) 
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3. Approximately what percentage of the students attending your school are children 
of 

% A Professional or managerial personnel 

% B Sales, cler.cal, technical or skilled workers 

% C Factory or other blue collar workers 

% D Farm workers 

% E Persons not regularly employed 

X F Persons on welfare 

( Items -A-F should add to 100%) 

100% 

4. Approximately what percentage of the students attending your school are 
% A American Indian or Alaskan Native 

% B Asian or Pacific Islander 

% C Hispanic, regardless of race (Mexican, Puerto Rican, Cuban, Central 

~ ~ ~ or South American or other Spanish culture or origin) 

% D Black and not Hispanic 

% E White and not Hispanic 

(Items A-E should add to 100%) 

100% 

5. Dot»s your school H ualify for ESEA Title I assistance*? 

Yes - If Yes* approximately what number of students qualify for 
and what number of students are receiving ESEA Title I assistance? 

Approximate number of students qualifying for ESEA 

Title I assistance 

Approximate number of students receiving ESEA Title I 

assistance 

No 

THANK YOU FOR YOUR COOPERATION 
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Supplementary Principalis Questionnaire 
for 9- and 13-Year-01ds 



Instructions The purpose of this questionnaire is to provide 
additional information which wiii be used in the 
analysis of the Reading, Literature data Please 
answer the following- questions for each school 
grade listed Darken the appropriate ovals with 
a soft lead pencil If you have questions about 
any of the following items, please contact the 
National Assessment District Supervisor Thank 
you fur vour cooperation 



Does >our school have a school library' 

O Yes (Go to Question 2 on page 2 I 
cr> So (Go to Question 4 on page 3 } 

If your school has a library, approximately how many volumes are housed 
in it? 



If your school has a library, * hen are your library facilities available for the 
use of students? 



A. 


Before school hours 




Yes 


o 


No 


B 


During school hours 


<=> 


Yes 


c=> 


No 


C 


After school hours — tfternoon 


o 


Yes 


cz> 


No 


D 


After school hours — night 


CD 


Yes 


c=> 


No 


E 


Weekends 




Yes 


cz> 


No 
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Whether or not your school has a library, are any of :he following sources of 
reading materials available to students in your school 9 



A 


ridmdua! classroom collections 
of reading materials 


cd 


Yes 


CD 


No 


B 


Bookmobile 


CD 


Yes 


CD 


No 


C. 


Cooperative programs with libraries 
outside the school 


cd 


. es 


CD 


No 


D 


Book c'**bs 


CD 


Yes 


CD 


No 


E. 


Book store or book trading post 


CD 


Yes 


CD 


No 


F 


Books donated by individuals or 
parent organisations 


CD 


Yes 


CD 


No 


G 


Students' personal book collections 


CD 


Yes 


CD 


No 


H 


Other 

Please specify 


CD 


Yes 


CD 


No 


I 


Other 


CD 


Yes 


CD 


No 



Are any of the following instructional methods or materials available to help 
students in your school read better? 



A 


Individualized reading materials 


CD Yes 


O No 


B. 


Audio-visual aids (e.g., tape recordings, 
film strips) 


CD Yes 


CD No 


C. 


Computer-assisted instruction 


o Yes 


CD No 


D, 


Teaching machines (e.g., controlled 
reader, tachistoscope) 


CD Yes 


CD No 


E. 


Skill exercises or worksheets (e.g.. 
vocabulary, comprehension, study skills) 


cd Yes 


CD No 


F 


Heading games, either commercial or 
teacher-made 


cd Yes 


CD No 


G 


High interest/low vocabulary level 
reading materials 


CD Yes 


CD No 


H 


Other 

Piease snppifv 


cd Yes 


CD No 


1 


Other 

Please sppnfv 


CD Yes 


. cd No 
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6. Approximately what percentage of your entire teaching stiff (including 
teachers in areas such a? art. music, physical education, etc ) have received 
in-service training in reading from your school or district' 

CD In-service training not offered 

CD 1-9% received tra.ning 

CD 10-24% received training 

CD 25-49% received training 

CD 50-99% received training 

CD 100% received training 

1 Does >our school pro\ ide separat e classes in remedial reading (i.e.. a 
"PMihgu t" program in contract to additional assistance in the regular 
classroom setting) 9 

CD Ye* iGo to Question 8 on page 6 ) 
CD No (Go to Question 1 1 on page 8 i 

8. If separate remedial reading classes are offered, approximately what 
percentage of your students enrolled in the following grades participate 
in these classes? 

No such 







grade 


0% 


1-9% 


10-24% 


25 49% 


50-99% 


100% 


A. 


Crude I 


CD 


CD 


CZ> 


CD 


CD 


CD 


CD 


B. 


Grade 2 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


C. 


Grade 3 


CD 


CD 


i.. .1) 


CD 


CD 


CD 


CD 


D. 


Grade 4 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


£. 


Grade 5 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


F. 


Grade 6 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


G 


Grade 7 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


H 


Grade 8 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


1. 


Grade 9 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


<J 


Ungraded 


CD 


CD 


CD 


CD 


CD 


CD 


CD 
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If separate remedial reading classes are offered, do any of the following 
persons teach or assist with these classes' 



A 


Certified reading teacher or specialist 


<=3 Yes 


<=o No 


B 


Classroom teacher with no special 
certification in reading 


co Yes 


co No 


C 


Paraprofessiohal or paid reading aide 


CO Yes 


co No 


D 


Other 

Please soeci'y 


co Yes 


co No 


£ 


Other 

Please soeeifv 


a Yes 


co No 



If separate remedial reading classes are offered how frequently are the 
follow ing methods used to ,dent ifv students for these classes' 









Often 


Sometimes 


Never 


' A 

L_ 


Group administered standardized 
reading tests 




CO 




CO 


! B 

1 


Individually administered 
reading tests 


co 


CD 


CO 


CO 


r^- 


Teacher referral 




CZ> 


CO 


o 


\± 


Counselor referral 


CO 


CO 


CD 


CO 


1 K 


Parent reques? 


CZ> 


CD 


( — > 


CO 


K 


Self referral 


o 


CD 


CD 


CO 


<; 


Other 

Please *penf* 




< ) 


CO 


CO 




Other 


CZlt 


CD 


CD 


CO 



Please specifv 
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11, Does your school offer se parate enrichment reading classes designed to 
improve reading ski lis of good readers' 

cd Yes (Go to Question 12 ) 

cz> No (Stop Do not answer Question 12.) 

12. If separate enrichment reading classes are offered, approximately what 
percentage of your students enrolled in the following grades participate in 
these classes 7 







No such 
grade 


0% 


1-9% 


10-24% 


25-49% 


50-99% 


100% 


A 


Grade 1 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


B 


Grade 2 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


C. 


Grade 3 


cd 


CD 


CD 


CD 


(ZD 


CD 


CZ) 


D 


Grade 4 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


E 


Grade 5 


cz 


CD 


CD 


CD 


CD 


CD 


CD 


F 


Grade 6 


CD 


CZ) 


CD 


CD 


CD 


CD 


CD 


G 


Grade 7 


CD 


CD 


CD 


CD 


CZ) 


(ZD 


CD 


H 


Grade 8 




< — > 


( > 


< — J 


( ) 


C ) 


CZ) 


I 


Grade 9 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


J 


Ungraded 


CD 


CD 


CD 


CD 


CD 


CD 


CD 



THANK YOU FOR YOUR COOPERATION 
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Supplementary Principal's Questionnaire 
for 17-Year-01ds 



Instructions The purpose of this questionnaire is to provide 
additional information which will be used in the 
analysis of the Reading/Literature data Please 
answer the following questions for each school 
grade listed Da-ken the apipropriate ovals with 
a soft lead pencil If you have questions about 
any of the following items, please contact the 
Nation*! Assessment District Supervisor Thank 
you for your cooperation 



1 Does your school have a school library 7 

o Yes Hio to Question 2 on page 2 ) 
CD No (Go to Question 4 on page 3 ) 

2 If your school ha« a library, approximately how many volumes are housed 
in it? 



3 If your school has a library, when are your library facilities available for the 
use of students' 



A 


Before school hours 






Yes 


CD 


No 


B. 


During school hours 




o 


Yes 


CD 


No 


C 


After school hours — 


afternoon 




Yes 


o 


No 


D 


After school hours — 


night 




Yes 


CO 


No 


E. 


Weekends 




CD 


Yes 


o 


No 



i 
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4 W.hether or nut your school has a libra r> are an\ uf the follow ,ng sources of 
reading materials available to students in your school ' 



I A Indi\ idual classroom collections 
of reading materials 

| B Bookmobile 



co \ es 
co Ves 



(* t'ooperatiNe programs w 1 1 h libraries 
outs id** the school 



o Ye-, 



j f) Book dubs 

j K Book store or book trading post 

' F Books donated bv individuals or 

1 parent organizations 



r 



i\ Students' personal Look collections 



CO Yes 



i H Other 

Please specif > 



Yes 



LJ No 
TO So 



co No 



co Y« s co No 

co \ es co No 

CO Yes CO No 



No 



^ No , 



! i Other 

! Please spt*eif\ 



Yes 



No 



5 Arc am of the fojlow mg instructional methods or materials a\ ailable to 
help students n your =i-hool eat! tetter'* 



A In(lividuali?ed reading materials 



' Yes co No 



B. Au']lt*-\'i>ua' .lids, {e g , tape recordings, co Yes 
film straps! 



C Computer-assisted instruction^ 

D Teaching machines (c g , controlled 
reader, taeh otoscope) 



F Reading games either corn men tal o»* 
teacher made 



High interest *<>a vo* .tlmlar \ level 
reading materials 

h Other 

Please spet if\ 



Other 

Plea e spee|f\, 



co Yr« 



K Skill exereises or w orksheets (e g , CO 
voi abulary, comprehension stud\ skills) 



co No 

co No 
co No 

co No 



\ es o No 

" No 



o No 



NY 
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Approximately what percentage of your entire teaching staff (including 
teachers in areas such as art. music, physical education, etc } have received 
in-service training m reading from your school or district' 

CD In-service training not offered 

CD 1-9% received training 

CD 10-24% received train ng 

cd 25*49% received training 

cd 50-99% received training 

cd 100% received training 



How many semesters (approximately 18 weeks) of English instruction are 
required in each of the following grades in your schcol 7 If your school is 
organized by quarters (approxirm.iely 9 weeks), please convert your answer 
to semesters t 1 quarter = \ semester) „ 







No such 
grade 


\ 

Semester 


1 

Semester 


1% 

Semesters 


2 

Semesters 


A 


Grade 9 


o 


CD 


<zz> 


cd 


CD 


B 


Grade 10 


o 




CD 


cd 


CD 


C 


Grade 11 


<zz> 


CZD 


CD 


CD 


CD 


D. 


Grade 12 


cd 


CZD 


CD 


CD 


CD 




J 



How arc the following topics r <_luded in your school's English or Reading 
curriculum 7 Fill in anly one oval for each U/ptc If a topic is. both a u nit of 
study and integrated into a general Kngh^h course, f * 1 1 in the oval under 
"concentrated unit of study" 



As a i ont filtrated 
unit of ttudi 



A Literature (by historical period) 

B Literature (by genre or interest 
grouping) 

C Stud., about language (e g , 

semantics, histor> of language) 

D Humanities (interdis* ipiinarv 
studies including literature 
and other subjects) 



E Composition 

F Creative w nting 

0 Speech 



H Debate 
I Theater 



X Journalism 

K M^ss media (radio, film TV* 



Study skills 



M Vocabulary development 
N Grammar 



0 



Other 

Please specify 



As a topic 
integrated 



of y v^t'fhs 


into a gen- 


Not 


or more 


eral English 


at 




course 


all 




C j 


O | 






- - H 
C_J) i 






| 




<_z> 


\ 




( 1 


1 

o 1 



- - 1 

O 



J 



P Other i_j (__, 

Please speeifv . 

? Does vour school offer an "honors" program or advanced placement in 
English'' 



Yes 
Sn 
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10 Does your school provide separate classes in remedia' reading (ie,i 

o 

" pull-out" program in contrast to additional assistance m the regular 
classroom setting)* 

CD Yes (Go to Question ii.) 

czd No (Go to Question 14 on page 10.) 

1 1 If separate remedial reading classes are offered, approximately whai 
percentage of your students enrolled in the following grades participate 
in these classes 9 







No such 
grade 


0% 


1-9% 


10*24% 


25-49% 


50-99% 


100% 


A 


Grade 9 


CZD 


CZD 


CZD 


cd 


CD 


CD 


CD 


B 


Grade 10 


czd 


CZD 


CZD 


o 


CD 


CZD 


CD 


C 


Grade 11 


CD 


czd 


CZD 


CD 


CD 


CD 


CD 


D 


Grade 12 


CZD 


CZD 


CD 


CD 


CZD 


CD 


CD 



12. If separate remedial reading classes are offered, do any of the following 
persons teach or assist with these classes? 



A. 


Certified reading teacher or specialist 


CD 


Yes 


CD 


No 


B 


Classroom teacher with no special 


CD 


Yes 


CD 


No 




certification in reading 










C. 


Paraprofessional or paid reading aide 


CD 


Yes 


CD 


No 


D 


Other 


CD 


Yes 


CD 


No 




Please specify- — — 






















E 


Other 


CD 


Yes 


CD 


No 




Please specify 

i. 
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13 If separate remedial reading classes are offered, how frequently are the 
following methods used to identify students for these classes 7 







Alwavs 


Often 


Somt times 


Never 


A 


Group-administered standardized 
reading tests 


CD 


a 


CD 


CO 


B 


Individually-administered 
reading tests 


co 


CD 


CD 


CO 


C 


Teacher referral 


o 


CD 


CO 


CD 


D 


Counselor referral 


< — ) 


CO 


< — ) 


i — i 


£ 


Parent request 


CO 


CD 




CO 


F 


Self referral 


CO 


CD 


CD 


CO 


G 


Other 

Please specify 


CD 


CD 


CD 


CD 


H 


Other 

Please specify^ 


CD 


CO 


CD 


CD 



14. Does your school offer separate enrichment reading classes designed to 
improve reading skills of good readers 7 



CD Yes (Go to Question 15 ) 

CD No (Stop, Do not answer Question 15 ) 



15. If separate enrichment reading classes are offered, approximately what 
percentage of your students enrolled in the following grades participate 
in these classes? 







No such 
grade 


0% 


19% 


10-24% 


25-49% 


50-99% 


100% 


A 


Grade 9 


CD 


CO 


CO 


'CD 


CD 


CD 


CD 


B 


Grade la 


CO 


CD 


CD 


CD 


CO 


CD 


CD 


C 


Grade 11 


CO 


CO 


CD 


CO 


CO 


CD 


CD 


D, 


Grade 12 


CO 


CO 


CO 


CD 


CO 


CO 


CD 



THANK YOU FOR YOUR COOPERATION 
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Package Cover Sheet 



2" 
4 

s" 

is" 
I™ 
2" 
4" 

8* 
l" 
2" 



A 
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* 

* 
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Approval Expires 12-81 
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not required to respond, sour cooperation is needed" to make the-xesults 
of thissurvev comprehensive, accurate and timely. 



PACKAGE 






i. 


I). 
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NTMBKK 


0 


2 












CZ3 


mm 




CLT-> 




C ^ 


f — J 


mm 


C_3 






c:o 




CO 


mm 


(__> 








C" „ 




CZD 


mm 




CO 


CO 


t~ ' > 


<CO 


o 


c_.> 




c: j 









8 

4* 

2" 

1 

Ti" 

8* 

4 

2" 

i 



co ;\ 
cr> P 
CO NR 
O NK 



H 



C2-J ■ } 

r - ' 1 

* Grade 2 or lower 
** Grade 6 or higher 



□ 

CO 
CO 



CO 

CO 
CD 



YEAR 11 
AGE CLASS 1 
GROUP 
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Directions to Exercise Administrators for 
Coding Package Cover Sheet 



Codes To Be Used in Columns 3-8 



Package Administration 
Coding Schedule 
Column Column Item 

3 3 Grade. 



Code 
Two digits: 

8th grade = 08 
Ungraded class = 98 
Special education 
class = 99 



Sex 



1 = Male 

2 = Female 



Bi rthdate 



Race 



Month and last two 
digits of year: 
May 1966 = 0566 

1 = W(white) 

2 = B(Black) 

3 = ^{Spanish Heritage) 

4 - I (American Indian 

or Alaskan Native) 

5 = A(Asian or Pacific 

Islander) 

6 = U (Unclassified) 



EA Number 



PSU and 

School 

Numbers 



Two digits; number 
recorded on the front 
cover of your manual 

Five d igi ts; 

First two = PSU 

Numbe r ; 
Last three = School 

Number ; 
as shown on the 
Administration 
Schedule 
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Standard Background Information Form for 9-Year-01ds 

! Miut l.imih j*' t ti m tp<*' r**uul.*t'\ ' 

^ ,.>, \o I don ' know 

J |;«m > \<»ur famiU > an\ tnag.i/im *. " mil.irh 

\«. 1 (|i»n • I- n<>A 

\ \r*-thir« moo shiin J"» hiHits in Mr'i^ honir ' 

\ * , \,> 1 d».n ( hnfA 

j j„ * hi f. i 'i i pi it i<i|n 'liit in \ i iii r hum* 1 1 

\, s \m i d«.n ; hih'a 

||.,\\ »n'ii h m ho.. I did M-iir '.if hfr t ompleie*' 

i] || | |\ mi 0\M)\A1 *Ahn h h«*v Oiuas hn»\ imnh m hool \uur 
; *h« r i ■ ■mpli ,ft d ' 

Did < - mipif o i hi* Hi h 'Jt-i'h 

t .,-?ipU ?i •] iKv Hih ^i* bin not g< } I" bigh m hool 
V.i-ir ♦<> I luh >m 5 <»m1 hu* did no* _j,idua*» nor i high ^f\ooi 
< ir.i'iu. >*«•(! »pim huh !m<>l 

Sonif .<if.»n * ! o*r nr,iiiu.i'inn trom high sc hool 

i doll J kilo A 

i, !>..! . |r , t xj.i/t 'ton, . < «'!j f hi. itr tini\i^s|f\ 1 

7 How rnuih school did \ our mother complete 0 

i FILL IN THK ONE OVAL which best shows how much school your 
mother c ompleted ) 

C." > Did not mmpleto the 8th grade 

d" Completed the Hth grade, but did not go to high school 

/ \Wnt to high si hool. but did not graduate from higb school 

r^-, ( tr.idtidted from high school 

_ S*itn«* «*dm at inn after graduation from high m hooi 

} don f b"A 

H !);} ..urn ,*r p'uc.'i 'f-ni a « ***** e» or tintwrMf m * M 

"j f . \ » I *ii*f know 
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Standard Background Information Form for 13-Year-01ds 



Dot's vuur t.imih a newspaper rt*i*nl«trl\ "* 
H-s \o ! (h>p'l kn<>v\ 

1>k*s vuur f«irnil\ am mittv/uu s h uul.trk*' 

\ i's N*. I di (*} i know 

XriMhiTi* rn«»r«' Mian 2~> books *n voitr hoine*' 
i ^ t«s \» > 1 don ! I mow 

There .hi I'll* » i lopcdia m \«*nr homi 1 '* 
N I'N \m I f|.»n t ^nW 

nun h s< hool ci id sour osfher * mpl«M t*** 

I III. UN Till- 0\K H\ \i *huh besi shows how nmch scho^i sour 
tat h<- r i « »r»iph'irH } 

Did not t nmpU'ti 1 ilu* 8th ^radc 
. ( ompk*t<»<l th< Mi h urath 1 hu* did not ti ( , io hi^h school 
j~, Wffil Ni hi^h school hut did not graduate from high school 
_ (<rariua!(d fron 1 hi^h ^ hoof 

Some i*Hu« tit ii hi alter uradu.'tiop imm high school 

f don't kiv»'A 

Did vour t.i'ftt-r LT,tdu,i f « tniu* mi- *j» umwrsit \ 
t - \o 1 don 1 mioa 

How mm h hooj d{fj vour m .iJu r • 'implctr'' 

I I ILL i\ 1 1 IK ONI* f)\ Al a^h >• h. si >hnws ho* much school \our 
mother < i.mplfh'ft I 

Did n«'f ' ornplt *e 'he Sih u r ad*» 

f (iinplfird fht- 8th ^r.uU , h?r did n»n ^o to hs^h s<hool 
Ut n' to high s> hooi T hu' did not graduate from high school 
i iraHiht'i fi horn hs^h m hoof 
Som«* * dm .itnm <»tti r ^rafJuatmn Inm hmn school 

I d> >n ' i-ri'i a 

I ht\ \ ' ill: r.< >' 1 .« » » •( hi i'« ' * • »■ .« » « dl< *i< < »* 'ini.c*rsp \ ' 

t «*s \i 1 d <h ? r II'iW 



-ni-o • - ho -.5! . .hi \ hu*nd*i\ 

It '»,« 5 'v ■ d <» i-« - • I'h Mniiti '!•< ,taf< ttr icrn'orv > 



1 > i' id' ■ ■ i •*! ■ ■ l . j -i» ( i* \ n* 
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Standard Background Information- Form for 17-Year-01ds 



1 How much school did your father complete'' 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed ) 

cz> Did not complete the 8th grade 

CD Completed the 8th grade, but did not go to high school 
cz> Went to high school, but did not graduate from high school 
CD Graduated from high school 
CD Some education after graduation from high school 

cd I don't know 

2 Did your father graduate from a college or uni\ersit\ 
CD Yes cz-j No ^_ I don't know 

3, How much school did your mother complete 7 

(FILL IN THE ONE OVAL which best shcwshm much .<hool your 
mother completed.) 
CD Did not complete the 8th grade 

i — > Completed the 8th grade, but did not go to high school 
CD Went to high school, but c*id not graduate from high school 

Graduated frem high school 
CD Some education after graduation from high school 

cd I don't know, 

4. Did your mother graduate from a college or univcr-it /' 
t — > Yes CD No - * l ^ uv - 1 kf! ^ u 

5 Where did you live on \our ninth birthdav ' 

CD In tht- I filled States (Please spenfs the -tate or territory ) 



Outside of the I'm ted States ( Please sporif\ the countr\ ) 



• I don't know 
Where did sou live on vour tmrteenth hirt hdnv 
C. > In the I'mtef 1 States {Please ^>ecif\ the state or territory I 



czr; Outside of tne l' ruled State- < Please -pe< if\ the country ) 



cr - ' 1 don't krnnv 
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Supplementary Background Information Form for 9-Year-01ds 



1. A. How important is it to be able to read? . * 

c=> It is very important. 

cz> It is somewhat important. 

c=> It is not important at all. 

B. How much do you enjoy reading? 

cz> I enjoy it very much. 
C3 I enjoy it somewhat. 
I do not enjoy it at all. 

2. What kind of reader do you think you are for your age? 

CD A poor reader 
cz> A good reader 

i 

o A very good reader 
cz> I don't know. 
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23. A. How much television did you vvarth yesterday? 

(—> None CD 2 hours o 5 hours 

CD 1 hour or less o 3 hours o 6 hours or more 

o 1 hour o 4 hours 

B. How much time did you spend reading just for your tnvn enjoyment 
yesterday? 

o None CD 2 hours o 5 hours 

i — > i hour or less cd 3 hours cd 6 hours or more 

cz> 1 hour cd 4 hours 

C. Did you go to kindergarten before you attended first grade? 

o Yes 
CD No 

D. Is English the language spoken most often in your home? 

o Yes 
cz> No 

E. Is a language other than English spoken in your home? 

CD Often 

CD Sometimes 

CD Never 
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Supplementary Background Information Form for 13-Year-01ds 



1. - A. How important is it to be able to read? 

cz> It is very important. 

o It is somewhat important. 

o It is not important at all. 

B. How much do you enjoy reading? 

CP I enjoy it very much, 
o I enjoy it somewhat, 
a I do not enjoy it at all. 

22. What kind of reader do you think you are for your age? 

(ZD A poor reader 
o A good reader 
o A very good reader 

o I don't know. 
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24. A. 



How much time did you spend on homework yesterday? 



cd> No homework was assigned, 

o I had homework but didn't do it. 

o Less than one hour 

C3 Between 1 and 2 hours 

CD) More than 2 hours 

B. How much television did you watch yesterday? 

CD) 5 hours 
CD) 6 hours or more 



CD) None cd> 2 hours 

cd> 1 hour or less CD) 3 hours 

cd> 1 hour CD) 4 hours 



C. How much time did you spend reading just for your own enjoyment 
yesterday? 



CD) None cd> 2 hours cdd 5 hours 

CD) I hour or less CD) 3 hours cd> 6 hours or more 

CD) 1 hour cd> 4 hours 



D. Is English thejanguage spoken most often in your home? 

CD) Yes 
CD No 

E. Is a language other than English spoken in your home? 

CD) Often 

CD) Sometimes 

CD) Never 
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Supplementary Background Information Form for 17-Year-0lds 



22. A. How important is it to be able to read? 



It is very important. 
CZ) It is somewhat important. 
CZ) It is not important at all. 
B. How much do you enjoy reading? 

cz> I en 1 it very much, 
czz) I enjoy it somevhat. 
<zz> I do not enjoy it at all. 

23. A. What kind of reader do you think you are for your age? 

c — s A poor reader 
CZ) A good reader 

A very good reader 
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CZ) I don't know. 
B. How would you classify the primary emphasis of your high school 
course**' 

CZ) General only 
t — > Vocational, technical, or trade 
cz> College preparatory 

( 3 1 don't know. 
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10. A. How m^c.t time did you spend on homework yesterday? 



ERIC 



CD No homework was assigned. 

CD I had homework but didn't do it. 

cd Less than one hour 

a Docween 1 and 2 hours 

CD More than 2 hours 

B. How much television did you watch yesterday? 



CD None . o 2 hours CD 5 hours 

c~* 1 hour cr less CD 3 hours CD 6 hours or more 

CD 1 hour CD 4 hours 



C. How much time did you spend reading just for your own enjoyment 
yesterday? 



CD None o 2 hours CD & hours 

CD 1 hour or less a 3 hours CD 6 hours or more 

CD 1 hour CD 4 hours 



D. Is English the language spoken most often in your home? 



Yes ° 



O No 



E.. Is a language other than English spoken in your home? 



Often 
CD Sometimes 
o Never 
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11. What is \our racial background? 

czd American Indian or Alaskan Native 

<zz> Asian or Pacific Islander 

czd Black 

( — ) White 

czz> Other(Please.speeify) , 



12. I \our ethnic heritage Hispanic (sueti as Mexican. Puerto Rican, Cuban, 
( entral or South American or other Spanish Culture or origin)? 

czd Yes 
czd No 



13. Which of the following does your family have at home? (Fill in one oval on 
each line.) 



Have Do not have 
I A. Newspaper received regularly CZD CZD 



B. Magazines received regularly czd czd 

(\ More than 25 books o czd 

I). Encyclopedia CZD CZD 

I K. Dictionary czd 



| F. Record player czd czd 



(i. Tape recorder or cassette player czd 

H. Typ< v\ riter czd 

\. Vacuum cleaner CZD 

J. Kleetrte dishua>her czd CZD 

K, Two or more cars or trucks that run cd gzd 



ERLC 



61 



Reading Background Questionnaire for 
17-Year-0lds Out of School 



1. A. In high school, did you learn the kinds of skills you now need in your 
everyday life? 

cz> Yes 
<=> No 



B. If no, what skills do you think you should have lea. ned in school? 



2. Are you now employed? 

<Z3 Yes, full-time (Go to Question 3 on page 4.) 
o Yes, part-time (Go to Question 3 on page 4.) 
cz> No (Go to Question 6 on page 5.) 



ERLC 
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7. 



3. 



If you are now employed full-time or part-time, how often do you use the 
following reference sources for your job? 



ERLC 



A. 
B. 



C. 
D. 



At least 
once a 
day 



At least 
once a 
week 



Telephone directories 



Catalogs 



Dictionaries 



Technical references (such as 
manuals on how to operate and 
maintain equipment, other "How 
to-Do" guides) 



Less than 
once a 

week Never 

l 



4. If you are now employed full-time or part-time, how often do you read the 
following types of materials for your ;oh 9 



Notes, letters, or memos 



B Forms (such as work muors, job 
orders, vouches, claims, or 
purchase orders) 



C\ Charts, graphs, maps, or tables 



D, Manuals on company rules, 
policies, and procedures 



At least 
once a 
dav 



CZD 



CT3> 



At least 
once a 
week 

CZZ) 



Less than 
once a 

week Never 



O CD 



o a 



<ZZ> CZZ) 
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If you are now employed full-time or part-time, how often do you have 
problems doing the reading required for your job? 

cd Often 

CD Sometimes 

CD Never 

A. Are you now enrolled in a school or training program? 

CD Yes, on a full-time basis (Go to Question 6B. below.) 
cd Yes, on a part-time basis (Go to Question 6B, below.) 
cd No (Go to Question 7 on page 6.) 

B. If yes, in what type of program are you enrolled? 

CD Primarily academic 

CD Primarily vocational or technical 

CD Other: 
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Look at the occupational categories shown cn the next page. They are num- 
bered 1 through 19. Choose the number of the ONE occupational category 
that best describes your main job and then fill in the oval beside the number 
you have chosen. 



o l 

o 2 

c=> 3 

o 4 

o 5 

o 6 

o 7 

o 8 

o 9 

CD 10 



o 11 

cz) 12 

cz> 13 

cz> 14 

cz> 15 

<=> 16 

cz> 17 

cz> 18 

cz> 19 - Other (Please describe.) 



CZJ I don't know. 



er|c 
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0C< CPATIONAL CATEGORIES 



1 - APPRENTICE - such as apprentice bricklayer, carpenter, mechanic, plumber 

2 - CLERICAL - Mich as bank teller, bookkeeper, secretary, typist, mail carrier, ticket 

agent . 

:> - ('RAFTER such as baker, automobile mechanic, machinist, painter, plumber, 
telephone installer, carpenter .. 

1 - FARMER. FARM MAN ACER . .. 

5 - FARM LABORER AND FOREMAN or FOREWOMAN 

0 - HOMEMAKER 

7 - LABORER - such as car washer, lumber millworker. sanitation worker, fisher ........ 

8 = MANAGER. ADMINISTRATOR - such as sales manager, office manager, school 

administrator, buyer, restaurant manager, government official 

9 - MILITARY - such as career officer, enlisted man or woman in the armed forces 

10 ~ OPERATIVE - such as meat cutter, assembler, machine operator, welder, taxicab, 

bus. or truck driver, gas station attendant 

1 1 = PROFESSIONAL - such a* accountant, artist, clergy, dentist, physician. reg : -tered 

nurse, engineer, lawyer, librarian, teacher, writer, scientist, social wo»"k*T. 
actor or actress 

12 = PRf )PRIETOR OR OWNER - such asou ner of a small business, contractor, rotaurant 

ou r.er 

I.'S PROTECTIVE SERVICE - <uch as defective, police officer or guard, sheriff, 
firefighter 

l i ~ SALES s'.K'h us sale^ worker, sale- c!e»*k. advertising <>r jnsuranep agent real •Mate 
broker . . 

1 .*> " SERVICE Mich ba^b^r |w>ai)t'>< k ian. practical nurse, prhate hn,>,-K»l'! \ t »'k'* 
jari! t «»r vvai'er or v\mtres> 

1 n it.* 11.11* ll/f'n.lll' i ,iunmuw« tiv'iinl! il* i.,m> 'ii f • * - « 

IT STI PKNT . . 

is •= r\f:\!PI,<>YKl> • 

1!» nTHKk 
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8. A. How important is it to be able to read? 

cz> It is very important. 

cz> It is somewhat important. 

cz> It is not important at all. 

B. How mu'th do you enjoy reading? 

cz> T enjoy it very much. 
cz> I enjoy it somewhat. 
c=d I do not enjoy it at all. 



9. A. What kind of reader do you think you are for your age? 

<td A poor reader 
cz> A good reader 
c=) A very good reader 

cz> I don't know. 



B. How would you classify the primary emphasis of your high school 
courses? 

cz> General only 

CD Vocational, technical, or trade 

ctd College preparatory 

czd I don't know. 
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10. A. How often do you read for your own enjoyment during your spare time? 



a Almost every day 

CD Once or twice a week 

CD Less than once a week* 

o Never 



B Which one of the following statements best describes the kind of 
reading you do in your spare time? 



CD I never read during my spare time. 

O I mostly read fiction; for instance, short stories, novels and plays. 

CD I mostly read nonfiction; for instance, bojks and articles about 
famous people, places, history, current events and hobbies. 

CD I read fiction and nonfiction about equally. 

How much television did you watch yesterday? 



o None 

CD 1 hour or less 

o 1 hour 



CD 2 hours 
o 3 hours 
CD 4 hours 



o 5 hours 

t — ) 6 hours or more 



B. How much time did you spend reading just for your own enjoyment 
yesterday? 



CD None o 2 hours c_, 5 hours 

CD 1 hour or less o 3 hours cz> 6 hours or more 

CD 1 hour cd 4 hours 
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C. Is English the language spoken most often in your home? 

o Yes 
a No 

D. Is a language other than English spoken in your home? 

cd Often 

o Sometimes 

czd Never 

12. What is your racial background? 

eo American Indian or Alaskan Native 

czd Asian or Pacific Islander 

CD Black 

czd White 

czd Other(Pleasespecifv) 



13. N your ethnic heritage Hispanic (such as Mexican, Puerto Rican, Cuban, 
Central or South American or other Spanish Culture or origin)? 

czd Yes 
o No 



9 

ERIC 



69 



14. Which of the following does your family have at home? (Fill in one oval on 
each line.) 







Have 


Do not have 


A. 


Newspaper received regularly 


CD 


cz> 


B. 


Magazines received regulaiiy 


CZ) 


cz> 


C. 


More than 25 books 


CD 


cz) 


D. 


Encyclopedia 


cz> 


cz> 


E. 


Dictionary 


cz> 


cz> 


F. 


Record player 


cz> 


CZ) 


G. 


Tape recorder or cassette player 


cz> 


CZ) 


H. 


Typewriter 


cz> 


CZ) 


I. 


Vacuum cleaner 


cz> 


CZ) 


J. 


Electric dishwasher j 

? 


CZ> 


CZ) 


K. 


Two or more cars or trucks that run 


(ZZ> 


CZ) 
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APPENDIX D 
RESPONSE RATES FOR ASSESSMENT SAMPLES 



Table I>1 shows the response rates 
for students assessed in 1970-71, 
1974-75 and 1979-80 • In the 1970-71 
and 1974-75 assessments, for each of 
the three age groups, 12 students and 
4 alternates .were selected for each 
assessment session. If all 12 stu- 



dents appeared for the session, then 
the alternates were dismissed. Other- 
wise, enough alternates were selected 
to bring the size of the group up to, 
or as near as possible to, 12. If 
the group assessed numbered between 8 
and 12 students, then the administra- 



TABLE D-l. Number of Students Assessed for 
Reading/Literature and Percent of Sample 
Covered, by Age and Assessment Year 



Year 


Age 


Type 


Nunber 


Total 


Average 


Average 






of 


of 


Number 


Number 


Sample 






Adminis- 


Packages+ 


of 


Assessed 


Covered 






tration 




Students 


Per 


in 










Assessed 


Package 


Percent 


1970-71 


9 


r, 


9 


21,220 


2,580 


90.7 


> 


13 


I 


3 


6,612 


2,204 


91.8 




G 


13 


33,202 


2,554 


88.2 




17 


I 


2 


• 4,396 


2,198 


89.6 




G| 


10 


23,727 


2,373 


75.2 






I 


2 


4,319 


2,160 


76.0 


1974-75* 


9 


G 


3 


21,697 


7,232 


87.5 




13 


G 


3 


21,393 


7,131 


83.7 




17 


G 


3 


19,624 


6,541 


69.7 


1979-80 


9 


G 


11 


29,103 


2,646 


90.1 




13 


G" 


15 


41,574 


2,772 


86.0 




17 


G 


14 


36,109 


2,579 


77.3 



+In the 1970-71 assessment, some booklets were administered to indi- 
vidual students using an interview (I) mode. The other booklets were 
administered to groups (G) of students. 

#In the 1974-75 arsessment, each booklet was triple sampled to in- 
crease the effective sample size. There were other booklets for other 
content areas also assessed in 1974-75; these are not included in 
this table. 



71 



erJc 



si 



tion was considered complete. If the 
final total was not at least a quorum 
of 8, a second and sometimes a third 
make-up session was held. The per- 
centages in Table 1>1 are based on 
the numbers of students assessed from 
the original groups of 12 selected 
and do not reflect the use of alter- 
nates. 

For the 1979-80 assessment, slightly 
different procedures were used. The 
nunber of students selected for each 
administrative session varied from 
16-25 students depending on the num- 
ber of eligible students and the pre- 
vious response rates obtained from 
schools in similar communities. No 
alternates were selected. The quorum 
size needed to consider an adminis- 
trative session complete varied ac- 
cording to the number of students 
selected. Since nonresponse rates 
have always t^een relatively small for 
ages 9 and 13 f the make-up or follow- 
up procedures used in 1979-80 for 
these ages were similar to the ones 
used for the 1970-71 assessment. If a 
quorum wife not obtained at the first 
administrative session, a second and 
sometimes a third make-up session was 
held. At age 17, in the 1979-80 
assessment, follow-up procedures were 
conducted on a school, rather than a 
session, basis. Tf a school had an 
overall tesponse rate pf less than 
75%, then all nonrespondents in the 
school were contacted for one or two 
follow-up sessions. These follow-up 
procedures for 17-year-olds provided 
satopLe coverage similar to that ob- 
tained at ages 9 and 13. 

Since response rates at age 17 have 



always been somewhat lower than at 
the other two ages, the Research Tri- 
angle Institute (RTI) , Raleigh, North 
Carolina, was asked to conduct a spe- 
cial study of nonrespondents during 
the 1972-73 assessment of science and 
mathematics. TTie results (Kalsbeek et 
al., 1975; Rogers et al., 1977) indi- 
cated that about 80% of the total 
nonrespondent group did not appear at 
the assessment sess i ons because o f 
conflicting school activities or ill- 
ness. Hie remaining 20% did not seem 
to be available. They attended school 
infrequently, if at all (for practi- 
cal purposes, they had dropped out) , 
or they had moved out of the school 
attendance area . In e ither case , 
these students probably should not 
have been listed in the in-school 
population of eligibles. 



Tables published in previous National 
Assessment reports showing response 
rates for age 17 generally conta in 
percentages adjusted to account for 
those 17-year-olds listed but not 
attending school. But, since National 
Assessment has not had the resources 
to replicate the RTI study in recent 
assessments, the 20% figure used as a 
basis for adjusting these percentages 
may be outdated and thus, the per- 
centages given in Table D-l have not 
been adjusted. It seems likely that 
despite efforts to update the lists 
of eligibles, these lists still con- 
tain some percentage of students who 
have i n e t f ect 1 eft che school s * 
Therefore, the percentages listed for 
age 17 are probably underestimates of 
the actual response rates for 17- 
year-olr's attending school. 
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APPENDIX E 



COMPUTATION OF MEASURES OF ACHIEVEMENT, 
CHANGES IN ACHIEVEMENT AND STANDARD ERRORS 



Several measures cf achievement that 
National Assessment uses in its re- 
ports are described in Chapter 7 of 
this docunent. The sample design, as 
described in Chapter 4, is a complex, 
deeply stratified, multistage proba- 
bility sample design. Measures of 
achievement are obtained by weighting 
individual responses appropriately. 
'Reasonably .good approximation of 
standard error estimates of these 
achievement measures can be obtained 
by applying the jackknife procedure 
to first-stage sampling units within 
strata, using the mc-rJiod of succes- 
sive differences and accumulating 
across strata. 

In thi* s^tion, the measures of 
achievement are first defined in al- 
gebraic form, followed by a descript- 
ion of the jackknife method that 
National Assessment uses to estimate 
the standard errors of achievement 
measures. 



Measures of Achievement 

Based on the sample design, a weight 
is assignee to eve*:y individual who 
responds to an exercise administered 
in an assessment. The weight is the 
reciprocal of the probability of se- 
lecting a particular individual to 
take a particular exercise with ad- 
justment for nonresponse. Since the 
probabilities of selection are based 
on an estimated number of people in 



the target age population, the weight 
for an individual estimates the nun- 
ber of similar people that individual 
represents in the age population. As 
explained in Appendix E, the weights 
are adjusted to reflect information 
from previous assessments on popula- 
tion distributions. 

A sum of the weights for all individ- 
uals at an age level responding to an 
exercise is an estimate of the total 
number of people in that age popula- 
tion. A sum of weights for all indi- 
viduals at an age responding cor- 
rectly an exercise is an estimate 
of the number of people in the age 
population who would be able to re- 
spond correctly if the entire popula- 
tion were assessed. These concepts 
also apply to any reporting group 
(e.g., defined by region, sex, and so 
on) and category of response (e.g., 
correct, incorrect and 11 1 don't 
know" ) . 

Let W fhk ~ sum °~ eights for respon- 
dents to exercise e who 
are in reporting subgroup 
j. and who are in the kth 
replicate of the hth sam- 
pling stratum, and 



hk = sum of Wei 9hts for respon- 
dents to exercise e who 
are in reporting subgroup 
U who are in the kth rep- 
licate of the hth sampling 
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stratum and who selected 
response category j. (e.g., 
correct foil) for the ex- 
ercise • 



portion in subgroup _i who would se- 
lect category j_ on exercise e and the 
proportion in the nation is denoted 
by: 



Note that WfU = Z C- 3 



'ihk 



ihk' 



(3) ^? ej - p e ^ - P e J 
i i A 



Then summing k over the h^ sample 

replicates in the stratum h, and sum- 
ming over the H sampling strata, 

H n t 



"ihk 



h=l k=l 

estimates *the number of eligibles in 
the population who are in subgroup i_. 



Similarly, C. 



i++ 



H 



h ei 
E C,/, est - 

w i i lhk 
h=l k=l 



mates the nunber of eligibles in the 
population who are in subgroup _i and 
who would select response category j_ 
for exerc.se e. 



National Assessment also reports f the 
n .r* thmetic mean of the percentage cf 
correct responses over sets of exer- 
cises corresponding to the measures 
in (1), (2) and (3). These, means are 
taken over the set of all exercises 
or a Subset of exercises classified 
by a reporting topic or content ob- 
jective. The mean percentages of 
correct responses taken over m exer- 
cises in some set of exercises corre- 
sponding to measures (1), (2) and (3) 
are, respectively: 



e 



An estimate of the proportion of the 
eligibles in the age population in 
group _i who would select response 
category j_ on exercise e is: 



(5) p, 



(1) p 



In the special case where the pro- 
portion of all age-el igibles who 
would select response category 2 on 
exercise e is estimated, the index A 
(for ALL) will be used in place of _i 
as follows: 



(2) p 



A-H- A++ 



In National Assessment reports, the 
proportion in (1) multiplied by 100 
is called the group percentage, and 
the proportion in (2) multiplied by 
100 is called the national percent- 
age. The difference between the pro- 



(6) 



LP, + P - P. 
i l A 



Note that the response category sub- 
script j_ has been suppressed since 
the means are understood to be taken 
over the correct response category 
for each exercise. 

Each of these six achievement meas- 
ures is computed and routinely used 
in reports describing achievement 
data for any assessment. The simple 
difference in these measures between 
two assessments of the same exercise 
(or sets of exercises) provides six 
measur es o f chang e in ach i evemen t 
that are routinely used in National 
Assessment's change reports. The next 
section describes how standard errors 
are estimated for the 12 statistics 
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used in NAEP reports. 

Computation of 
Standard Errors 

In order to obtain an approximate 
measure of the sampling variability 
in the statistics (1) through (6), a 
jackknife replication procedure for 
estimating the sampling- variance of 
nonlinear statistics from complex, 
multistage samples was tailored to 
National Assessment's sample design. 
Miller (1968, 1974) and Mosteller and 
Tukey (1977) provide information 
about the jackknife technique, wh^e 
Folsom (1977) describes how the pro- 
cedure is used in estimating standard 
errors for National Assessment's sam- 
ple design. 



ing standard errors, National Assess- 
ment replaces C^ k and W* hk from the 
hkth replicate with corresponding 

sums from another paired replicate in 
the same stratum. The replicate esti- 
mate is then computed. The replicate 
esthetes to be used in the calcula- 
tions are determined by arranging all 
the replicates in each stratum into 
successive pairs. That is, replicate 
1 is paired with replicate 2, repli- 
cate 2 with re'licate 3, 3 with 
4,...(n^-l) with n^ and replicate n^ 

with replicate l. 



To demonstrate the computational as- 
pects of this technique, consider 
estimating the variance of the sta- 
tistic^ in (1) — the proportion of 
age-el igibles in subgroup i who would 
select response category j on exer- 
cise e. 

This statistic is based on the data 
from all the n^ replicates in tho H 

strata. Let P^ hk be defined as a 

replication estimate of P e 3 and 

i 

constructed from all the replicates 
excluding the data from replicate k 
in stratun h. These replication esti- 
mates are computed as if the excluded 
replicate had not responded, and a 
reasonable nonresponse adjustment is 
used to replace the .data in replicate 
hk in estimating pp. Several choices 

for replacing the data in replicate 
hk are available. In order to obtain 
a convenient and computationally ef- 
ficient algorithm for approximat- 



ive contribution to the variance of 

P e ^ by each pair of replicates is the 

change in the value of the statistic 
incurred by replacing the data from 
each replicate in the pair with - the 
data from the other replicate in the 
pair and recomputing P e 3 i n the 

i 

usual way. This produces two repli- 
cate estimates. Squaring the differ- 
ence between these replicate esti- 
mates and then dividing by eight 

measures the contribution of this 
pair of replicates to the total vari- 
ance. The sum of these contributions 
over all successive pairs in the 

sua tun is the contribution by stra- 
tun n to the total variance. The 
square root of the sun of the H stra- 
tum contributions is the-estimate of 
the standard error of p " m 



75 



algebraically, the two replicate ss- 
t:~at3S for the pair <, k+1 f'.vner? 
<=I M ».^ and n,-l=L) are: 

l " I-*-* " ih k. lh ( k+ 1 ) 
'7) . - 

' ihk ' ih( k+1 * 

and 

c J - c^ J + c r 
"i*t- ih(k+l) ink 

e 1 ♦ 

{ o\ v ; = — 

5) 4 i-h(krl) ..ej _ ej + y ej 

The contribution to the total vari- 
ance from stratum h is: 

8 ^ i-hk i-h(k+l) / 

And finally, 3n estimate of the stan- 
dard error of P e ^ is: 

i 

H 4 
' :0) SE / P e: ) - l : var . 



Multiplying P e l by 100 yields the 

percentage of response to category 

Multiplying SE(P e] ) by 100 yields 

the corresponding estimated standard 
error of the percentage. 



In general , the jackknifed stanoard 
errors -*f the proportion estimates 
will be larger than the simple random 

sampling formula (PQ/N)^/ ? , 



where P=P e:l , 0=1-9 and N is the num- 
ber of sanpled respondents in sub- 
group ^ who took the exercise* The 

larger size of SE(P e ^) ref 1 ects 

i 

-nainly the loss of precision due to 

cluster-sampling of schools and stu- 
dents. The standard errors for the 
achievement measures (2) throuqh (^) 
are computed through a series of 
steps analogous to tnose followed in 

computing SE(P^) . 



The standard errors for the differ- 
ences between two assessments for any 
of the achievement measures (1) 
through (6) are computed as the 
square root of the sum of the squared 
standard errors from each of the sep- 
arate assessments. 

The size of the standard errors de- 
pends largely not only on the number 
of replicates and schools included in 
the sample, but also on the number of 
respondents* in each of the reportina 
groups. Table E-l shows the average 
number of students respond i ng to an 
exercise booKlet for each of the re- 
porting groups for each age for each 
assessment year. Table L-2 shows 
National Assessment's current esti- 
Hf^.es of the proportions of students 
in each rep rting group at each age. 
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TABLE E-l. Average Number of Respondents in Reporting 
Groups Taking an Item Booklet for All Three Age Groups 



Nation 

Region 
Northeast 
Southeast 
Central 
West 

Sex 
Male 
Female 

Race/ethnicity 
White 
Black 
Hispano 
Other 

Parental education 

Not graduated high school 
Graduated high school 
Post high school 
Unknown 

Type of community 
Rural 

Disadvantaged urban 
Advantaged urban 
Other 

Size of community 
Big cities 

Fringes around big cities 
Medium cities 
Small places 

Grade 

3, 7, 10 

4, 8, 11 

12 

Other 

Note. Group totdl^ may not equal 
onces* 



Age 9 Age 13 Age 17 

1979-80 1979-80 In School 

1979-80 

2,646 2,772 2,579 

628 690 604 

621 634 559 

735 775 750 

662 672 667 



1,330 1,364 1,300 

1,316 1,407 1,279 

2,028 2,144 2,090 

382 412 330 

190 176 124 

46 39 35 

ISO ?93 343 

673 M73 869 

1,007 1,314 1,280 

786 292 87 

266 274 247 

264 278 278 

263 274 260 

1,853 1,946 1,794 

58? 510 537 

450 552 434 

273 472 481 

1,340 1,238 1,128 

708 740 330 

1,873 1,954 1,981 

230 

64 78 38 



the nation due to round inq differ- 
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TABLE E-2. Estimated Current Population Proportions 
of National Assessment Reporting Groups 
for In-School Students 



Reporting Groups 

Sex 
Male 
Female 

Race/ethnicity 
White 
Black 

t Other 

Region 
Northeast 
Southeast 
Central 
West 

Parental education 

Not graduated high school 
Graduated high school 
Post high school 
Unknown 

Type of community 
Rural 

Disadvantaged urban 
Advantaged urban 
Other 

Size of community 
Big cities 

Fringes around big cities 
Mediun cities 
Small places 

Grade in school 
<3, <7, <10 

3, 7, 10 

4, 8, 11 
<4, <8, 12 
Other 



Age 9 . Age 13 Age 17 

.50 .50 .48 

.50 .50 .52 

.79 .80 .83 

.14 .13 .12 

.07 .07 .05 

.25 .25 .25 

.22 .23 .20 

.27 .27 .29 

.26 .25 .26 

.09 .13 .15 

.24 .32 .32 

.33 .42 .48 

.34 .13 .05 

.08 .10 .08 

-07 .07 .09 

.il .11 .11 

.74 .72 .72 

.20 .21 .19 

.22 .22 .26 

.12 .11 .11 

.46 .46 .44 

<.01 .02 .02 

.23 .25 .13 

.75 .72 .75 

<.01 <.01 .10 

<.01 <.01 <.01 
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APPENDIX F 



ADJUSTMENT OF RESPONDENT WEIGHTS BY SMOOTHING TO REDUCE 
RANDOM VARIABILITY OF ESTIMATED POPULATION PROPORTIONS 



Background 

As noted elsewhere, a weight is as- 
signed to e^ery individual who re- 
sponds to an exercise administered in 
an assessment. The weight is the re- 
ciprocal of the probability of selec- 
tion of the individual with adjust- 
ment fpr nonresponse. Hie weight for 
an individual estimates the number of 
people that the individual represents 
in the age population. The sum of the 
weights of all individuals at an age 
level who responded to an exercise is 
an estimate of the total number of 
people in that age population in the 
year that the exercise was assessed* 
Similarly, the sum of weights for all 
individuals who took the exercise and 
who also are members of some demo- 
graphic category (such as blacks) 
gives an estimate of the number of 
people in the age population, for the 
year, who are also members of the 
category. The ratio of the two totals 
estimates the proportional represen- 
tation of the demographic category in 
the age population for the given 
year. 

Separate estimates of the propor- 
tional representation of the various 
demographic subgroups are provided by 
each booklet administered to a 
particular age group in a given year. 
IXie to random sampling variability, 
the estimates of population propor- 
tions fo>* a given year based on sin- 
gle booklets will vary. There is also 
random sampling variation in esti- 



mates of population proportions from 
year to year in addition to whatever 
trends in population proportions over 
time that might exist. 

It is desirable to reduce the random 
variability of population proportions 
as much as possible since this varia- 
bility has an effect on performance 
estimates. For example, the percent- 
age of acceptable responses for an 
age group is a function of the rela- 
tive proportions of high-performing 
and low-performing groups. If the 
relative proportions of these groups 
are very different in different as- 
sessments due to sampling variabil- 
ity, then a portion of the change in 
percentage of acceptable responses 
for an age group could be attributa- 
ble to yearly sampling differences in 
the relative proportions of high- and 
low-achieving groups. 

In addition to reporting performance 
estimates for an age group as a 
whole, National Assessment also re- 
ports performance for various subpop- 
ulations, such as whites or blacks. 
Because variability of subgroups 
within these subpopulations (such as 
males and females within the white 
subpopulaf ion) influences the per- 
formance estimates for the 
subpofxilations, it is desirable that 
fluctuations of proportions of all 
subgroups cf each subpopulation be 
reduced ps much as possible. 

For each age and year, each of the 
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various booklets administered pro- 
vides estimates of a given population 
proportion. Since these estimates 
are subject to booklet-to-booklet 
variability, a better estimate of the 
population proportion, which will 
have reduced variability, is obtained 
by combining the information from all 
booklets. Hoover, these proportions 
vary from year to year due to random 
sampling variability or systematic 
differences in sampling procedures. 
mh even better estimate of population 
proportions for any single year can 
be obtained by smoothing the propor- 
tions over several assessment years. 
The word "smoothing" is used ^ere in 
the sense of fitting a smooth curve 
to a sequence of numbers by robust/ 
resistant procedures (Tukey, 1977) . 
Smoothing estimates of population 
proportions reduces a large portion 
of the sampling variability while 
preserving, as far as possible, ac- 
tual trends occurring in the age pop- 
ulation. 

After the population proportions have 
been smoothed, adjusted weights are 
derived for the assessed individuals 
.so that the pooulation proportions 
compjted using *:he adjusted weights 
are equal to the smoothed propor- 
tions. The adjusted weights are then 
used for all analyses. 



Smoothing Procedures Used by 
National Assessment 

The no^t direct way to smooth propnr- 

into nntualiy exclusive multiway 
cells on the 1 as is of their member- 
ship in oat^qori^s of various impor- 
tant variables and then to smooth the 
proprtions within each of the re- 
sult inq mulMway s^lls across years - 
Unf ortumto] y , this procedure tend r j 
to ffoduce a larg- nuuber oi c^IIs 
wi^h few pr»op]p and, consent ontly, 



quite unstable estimates of smoothed 
proportions. 

To circumvent this difficulty, Na- 
tional Assessment has utilized vari- 
ous smoothing procedures. These pro- 
cedures, which are all basically 
we igh t i ng-cl ass ad j ustments a ppl i ed 
independently to each age, ce de- 
signed to control, to varying de- 
grees, fluctuations in certain key 
subgroups while avoiding, as much as 
possible, instabilities due to small 
cells. 

Hie procedure used in 1979-80 has the 
following characteristics: 

1. It produces a single ad justed 
weight for each individual. 

2. It affords good control on the 
distribution of proportions of 
certain key variables. 

3. It tends to produce stabilicy of 
performance estimates. 

4. It is relatively easy to imple- 
ment. 

Even though adjusted veights using 
this procedure differ slightly from 
the corresponding adjusted weights 
from the other procedures that have 
pr ev i ousl y been em pi oyed , Na t i ona 1 
Assessment intends to use weights 
obtained using this procedure for all 
fiture analyses of data assessed in 
earlier years. This is simply because 
we believe weights obtained through 
this procedure to be the best availa- 
ble. 



The Current Smoothing 
Procedure 

The first, step in the soothing pro- 
cedure involved * he part- i Honing of 
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the population of age-class eligibles Table F-l. The same cells were used 
into the six smoothing cells given in for all ages. 



Cell 

1 

2 
3 
4 

5 



TABLE F-l. anoothing Cells Usee for the 
1979-80 Smoothing Procedure 



Race 

White 
White 
White 
Black 
Black 
Other 



Region 

All 
All 
All 

Southeast 
Not Southeast 
All 



Community Size 

Big city + fringe (BOFR) 

Med i un city (MC) 

Small places (SP) 

All 

All 

All 



Then, for each age and every year, 
the proportion of the population in 
each of the cells was estimated. For 
a given age and year, the proportion 
of the population in a particular 
cell was computed as the sum of 
weights of all respondents assessed 
in the given year who were of the 
specified age and who belonged in the 
cell, divided by the total of the 
weight of all respondents of the 
given aqe assessed in that year. 

Each of the six cells was corn prised 
of a sequence of estimated popilation 
proportions corresponding to the var- 
ious years of assessment. Each such 
sequence of proportions was then 
smoothed by fitting robust/resistant 
lines. Using data from the U.S. Cen- 
sus Bureau and the Current Population 
Survey, trends in enrollment by age 
and race and by age and region were 
obtained. The data from these surveys 
w^re adjusted to correspond with Na- 
tional Assessment definitions as much 
as possible. The resistant lines 
within the smoothing cells were con- 
strained to satisfy the trend from 
the U.S. Census and Current Popula- 
tion Survey data. 



The ^inal step in the smoothing pro- 
cedure was to adjust the respondents 1 
weights to be consistent with the 
smoothed proportions. Since each re- 
spondent takes only one booklet, the 
weight adjustments were done indepen- 
dently for each booklet. For a qiven 
age, year arid booklet, population 
proportions using the original 
weights were obtained for each of the 
smoothing cells. Then the weights of 
all respondents of a given cell were 
multiplied by the ratio of the 
smoothed cell proportion to the pro- 
portion using the original weights. 
This produced the adjusted weights 
th^.l were used in all analyses. 

Adjustment of Weights 
by Users 

The smoothed population proportions 
for 9-, 13- and 17-year-olds {in- 
school only) are given in Tables F-2, 
F-3 and F-4, respectively. TTie col- 
umns of each table represent the 
smoothing cells, while the rows rep- 
resent the assessment years, For ex- 
ample, the smoothed population pro- 
portion of 9-year-olds in soothing 
cell 2 (whites in medium cities) for 
1972-73 is 0.1152. 
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TABLE F-2. Smoothed Frequencies From 10-Year Smooth 
by Smoothing Cell and Year for 9-Year-01ds 





1 


2 


3 


4 


5 


6 


Race 


White 


White 


White 


Black 


Black 


Other 


Region 


All 


All 


All 


SE+ 


No": SE 


All 


Size of 












m T 1 

All 


Conmunity 


BC+FR# 


MC-H- 


SP## 


All 


All 


Year 














1970-71 


0.3299 


0.1203 


0.3574 


0.0557 


0.0736 


0.0631 


1971-72 


0.3232 


0.1177 


0.3647 


0.0562 


0.0743 


0.0638 


1972-73 


0.3165 


0.1152 


0.3720 


0.0568 


0.0749 


0.0646 


1973-74 


0.3098 


0.1126 


0.3793 


0.0573 


0.075* 


0.0654 


1974-75 


0.3030 


0.1101 


0.3866 


0.0579 


0.0763 


0.0661 


1975-76 


0.2963 


0.1076 


0.3938 


0.0584 


0.0770 


0.0668 


1976-77 


0.2896 


0.1050 


0.4011 


0.0590 


0.0776 


0.0676 


1977-78 


0.2829 


0.1025 


0.4084 


0.0596 


0.0783 


0.0684 


1978-79 


0.2762 


0.1000 


0.4157 


0.0601 


0.0790 


0.0691 


1979-80 


0.2694 


0.0974 


0.4230 


0.0607 


0.0797 


0.0698 



+SE = Southeast, 

#BC+FR = big cities + fringes. 

++MC = medium cities. 

##SP = snail places. 



To adjust respondent weights to be 
consistent with the smoothed propor- 
tions, the following procedures were 
followed: 

1. For each booklet, respondents 
were classified according to 
smoothing cell, and the raw popu- 
lation proportions for each cell 
were obtained. For example, the 
raw proportions for a booklet 
given to 9-year-olds in smoothing 
cell 4 was the total of the 
weights of all 9-year-olds re- 
ceiving the booklet who were 
black and in the Southeastern 
region, divided by the total of 
the weights of all respondents 
receiving the booklet. 

2. For erich booklet and smoothing 



cell, a weight adjustment factor 
as the ratio of the smoothed pop- 
ulation proportion (for the ap- 
propriate age, year and smoothing 
cell) over the raw population 
proportion was obtained. 

3. Ttie adjusted weights for an indi- 
vidual were the product of that 
individual's original weight and 
the appropriate adjustment fac- 
tor. 

Changes in Smoothed 
Proportions as New 
Assessments Are Completed 

Every t ime an assessment i s com- 
pleted, a new time point is added to 
each of the sequences of population 
proportions within the smoothing 
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TABLE F-3. Smoothed Frequencies From 10-Year Smooth 
by Smoothing Cell and Year for 13-Year-01ds 





1 


2 


3 


4 


5 


6 


Race 


White 


White 


White 


Black 


Black 


Other 


Region 


All 


All 


All 


SE+ 


Not SE 


All 


Size of 














Community 


BC+FR# 


MC++ 


SP## 


All 


All 


All 


Year 














1970-71 


0.3327 


0.1113 


0.3748 


0.0523 


0.0679 


0.0610 


1971-72 


0.3279 


0.1106 


0.3779 


0.0524 


0.0694 


0.0618 


1972-73 


0.3232 


0.1098 


0.3180 


0.0525 


0.0709 


0.0626 


1973-74 


0.3184 


0.1091 


0.3841 


0.0526 


0.0724 


0.0634 


1974-75 


0.3137 


0.1084 


0.3872 


0.0527 


0.0739 


0.0642 


1975-76 


0.3089 


0.1076 


0.3903 


0.0528 


0.0754 


0.0650 


1976-77 


0.3042 


0.1069 


0.3933 


0.0528 


0.0770 


0.0658 


1977-78 


0.2994 


0.1062 


0.3964 


0.0529 


0.0785 


0.0666 


1978-79 


0.2946 


0.1055 


0.3995 


0.053C 


0.0800 


0.0674 


1979-80 


0.2899 


0.1047 


0.4026 


0.0531 


0.0815 


0.0682 



+SE = Southeast. 

#BC+FR = big cities + fringes. 

+-WC = medium cities. 

##SP = small placco. 



cells. This means that, even though 
robust/resistant procedures are used, 
the addition of a new point may some- 
what change the values of smoothed 
proportions for prior years. Addi- 
tionally, any changes in methodology 
will impact the estimates. 

This means that the smoothed propor- 
tions, with the addition of the next 
assessment data, are apt to differ 
somewhat from the corresponding 



snoothed proportions without the new 
data. National Assessment has adopted 
the philosophy that the smoothed pro- 
portions, based on all currently 
available data using the best availa- 
ble algorithm, are the best availa- 
ble. Therefore, all subsequent analy- 
ses, for any year, will be done using 
this best-available information, even 
though this may produce estimates 
that slightly differ from prior val- 
ues. 
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TABLE F-4. Smoothed Frequences From 10-Year Smooth 
by Smoothing Cell and Year for In-School 17-Year-01ds 





1 


2 


3 


4 


5 


6 


Race 


White 


White 


White 


Black 


Black 


Other 


Region 


All 


All 


All 


SE+ 


Not SE 


All 


Size of 










All 


All 


Community 


BC+FRi 


MC++ 


SP## 


All 


Year 














1970-71 


0.3634 


t;. 1205 


0.3670 


0.0438 


0.0581 


0.0472 


1971-72 


0.3577 


0.1199 


0.3704 • 


0.0444 


0.0597 


0.0478 


1972-73 


0.3519 


0.1194 


0.3738 


0.0451 


0.0614 


0.0484 


1973-74 


0.3462 


0.1188 


0,3772 


0.0457 


0.0630 


0.0491 


1974-75 


0.3404 


0.1183 


0.3806 


0.0463 


0.0647 


0.0497 


1975-76 


0.3347 


0.1177 


0.3840 


0.0470 


0.0663 


0.0503 


19"?6-77 


0.3290 


0.1172 


0.3874 


0.0476 


0.0679 


0.0509 


1977-78 


0. 3232 


0. L L')t) 


0.3907 


0.0482 


0.0696 


0.0515 


1973-79 


0.3175 


0.1161 


0.3941 


0.0489 


0.0712 


0.0522 


1979-30 


0.3317 


0.1155 


0.3975 


G.0495 


0.0729 


0.0528 



+3F, = Southeast. 

#BOFR = big cities + fringes. 

++MC = nediun cities. 

##SP = small places. 
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GLOSSARY OF NATIONAL ASSESSMENT TERMS 



Acceptable response . Any response to 
an exercise that demonstrates 

^ achievement of the objective meas- 
ured by that e: ?rcise. 

Administration time . The total time 
allowed on the paced audio tape for 
an exercise. (Includes the time 
allowed for the stimulus and the 
response.) 

Administration timetable . Time peri- 
ods during the school year when the 
various age groups are assessed. 
The time periods are: 



October-December 
January-February 

March-May 



13-year-olds 
9-year-olds 
17-year-olds 



Age group or age level . Three age 
groups have been sampled in all 
three reading assessments : 9-year- 
olds, 13-year-olds, 17-year-olds 
attending school and also 17-year- 
olds vdio dropped out or graduated 
early. Birth date ranges fgr each 
age group in each of the tWree as- 
sessments are as follows: 

Assessment Age 9 Age 13 Age 17 

1970-71 1961 1957 10/53-9/54 

1974-75 1965 1961 10/57-9/58 

K79-80 1970 1966 10/62-9/63 

Assessment . The documentation of the 
progress in knowledge, skills and 
attitudes of American youth. Meas- 
ures are taken at periodic inter- 
vals for each learning area, with 
the goal of det^ 'ning trends and 
reporting the fii*uinqs to the pub- 
lic and to the education community. 



Assessment administrator . Individual 
employed to administer the assess- 
ment in participating schools. 

Background quest i ons . Questions 
about respondents' reading experi- 
ences in school and out of school 
were included. Standard background 
questions asked in every learning 
area are found on the back pages of 
the item booklets and include such 
things as level of parental educa- 
tion and reading materials in the 
home. Background questions used in 
the 1979-80 reading/literature 
assessment appear in Appendix C. 

Booklet . Items (exercises) are pre- 
sented to respondents in hnoklet-s. 
Booklets are designed to be scored 
by optical scanning machines. Each 
booklet contains (1) instructions 
for answering items and sample 
items, (2) assessment items and (3) 
background questions. Each booklet 
contains approximately 30-35 min- 
utes of assessment items and 10-15 
minutes of introductory material 
and background que^_ions. 

Category (scoring) . , A classification 
%>t a response to an open-ended ex- 
ercise. See scoring guide . 

Category within a var iable, A sub- 
classif l cation within a variable. 
For example, male and female are 
categories of the variable sex* t See 
reporting ^ groups. 

Difficul ty lev el. The percentage of 
acceptable responses to an exer- ' 
cise. 
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Exercise , A task designed to measure 
an objective. Because NAEP doe^not 
^minister "tests," but instead 
describes educational achievement 
over time, the term "exercise" is 
often used instead of the term 
"item" or "test item." The terms 
"item" and "exercise" are used syn- 
onymously in this report. 

Exercise booklet . See booklet . 

Exercise part . See item pert . 

Exercise pool . The "entire set of 
exercises prepared for a learning 
area. This set includes recycled 
exercises, exercises developed for 
previous assessments but not used 
due to exercise booklet or budget- 
ary constraints and newly developed 
exercises. 

Field test . A pretest of exercises 
to obtain information regarding 
clarity, difficulty levels, timihg, 
feasibility and spec iaKa^nun^ stra- 
ti ve problems needed for revision 
and selection of exercises to be 
used in the assessment. 

Grad^Htn school . Results are re- 
ported for 9-year-olds in' the 3rd 
and 4th grades; 13-year-olds in the 
7th and 8th grades; and 17-year- 
olds in the 10th, 11th and 12th 
grades. 

Group administration . Booklets were 
administered to groups of 10 to 25 
"students in 1979-80. Tn 1970-71 and 
1974-75, group size was 12 respon- 
dents. A paced audio tape was used 
to provide uniform instructions. 

Hand scori ng (scoring) . The coding 
of responses in a format compatible 
with the optical scanning equipment 
being usod. Multiple-choice exer- 
cises can be directly machine 
scored; however, responses to open- 



ended exercises must be coded in' 
scoring ovals so that they can then 
be machine scored ♦ See scoring 
guide . 

ID number . An identification number 
referring to the unique number as- 
signed to each respondent. This 
number is assigned to preserve the 
anonymicy of each respondent. NAEP 
does not keep records of the names 
of any individuals. 

Item . See exercise . 

Item booklet . Gee booklet . 

Item part . Each part of an item that 
asks a separate question. Parts 
may all pertain to one stimulus, 
such as a -g^aph or a table, or may 
concern the same topic. 

Jackknife . The name of t-he algorithm 
used by NAEF to estimate standard 
errors of percentages and other 
statistics. 

Learning area . One of the areas as- 
sessed by National Assessment: 
reading/literature, writing, mathe- 
matics, science, citizenship/social 
studies, art, music, career devel- 
opment. Also called "suoiect 



Level of parental educa tion. These 
levels are described in Appendix B. 

M odal grade . The grade in which the 
ma]ority of each in-school . age 
group is enrolled. For 9-year-olds, 
the modal grade is the 4th grade; 
fcr 13-year-olds, the 8th grade; 
and for 1 7-year~olds , the 1 1th 
grade. 

Objective . A desirable education 
goal agreed upon by scholars in the 
field, educators and concerned Lay 
persons, and established throiyh 
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the consensus approach. 

Qbj ec t i ves r ed evel o pnen t . After the 
initial assessment ol a learning 
a rea , one of the f i rst steps i n 
preparing for reassessment is a 
review of the learning-area objec- 
tives. This is carried out by 
scholars in the field, educators 
and concerned lay persons. These 
reviews may result in revision, 
modification or total rewriting of 
the learning-area objectives to 
reflect current curricular goals 
and emphases; they may also result 
in the endorsement of the objec- 
tives from the previous assessment 
as adequate for the next assess- 
ment. 

Open-ended exercise . A nonmultiple- 
choice exercise that requires some 
type of written or oral response. 

Paced audio tape , A tape recording 
that accompanies each booklet to 
assure uniformity in administra- 
tion. Instructions are played back 
from the tape recording so that 
reading difficulties will not in- 
terfere with an individual's abil- 
ity to respond. Response time is 
included on che tape. 

Primary samplin g unit (PSU) . First- 
stage sampling units, typically a 
county or a group of contiguous 
counties. 

Pr i no i pal ' s guest i onna i re . A data 
collection form given to school 
officials. The officials respond to 
questions concerning enrollments, 
size cf the community, occupational 
composition of the community, and 
so forth. Samples of these ques- 
tionnaires are found in Appendix C. 
See also supplementary principal's 
questionnaire . 

PSU . See primary sampling unit . 



Public-use data tapes . Computer tapes 
containing respondent-level exer- 
cise * and background/demographic 
data and machine-readable documen- 
tation. These tapes are available 
for use by external researchers 
wishing to do secondary analyses of 
National Assessment data. 

Racial/ethnic category . For the 

reading assessments, results are 

reported for whites, blacks and 
Hispanos (1979-80 only) . 

Receipt contro l . Procedures imple- 
mented by scoring staff to check in 
and screen materials from the 
field. Information gained from re- 
ceipt control procedures is relayed 
to assessment administrative staff 
so that any errors may be cor- 
rected. 

Recycled exercises . The set of exer- 
cises that is kept secure from one 
assessment to the next that will be 
used to measure changes (growth, 
stability or decline) in perform- 
ance for the learning area. 

Region . One of four geographical 
regions used in gathering and re- 
porting data: Northeast, Southeast, 
Central and West. States included 
in each region are shown in Appen- 
dix B. 

Re 1 ea sed exe r c i se . An exercise for 
which results and exercise text 
have been rep rted to the public. 

Released exercise set . A set of re- 
leased exercises, i ncl ud i ng docu- 
mentation and scoring guides, that 
can be purchased from National 
Assessment. Data for the released 
exeicise set are included as adden- 
dum pages. 

Reporting groups . Categories of var- 
iables tor which National Assess- 
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ment data are reported. Variable 
categories are defined in Appendix 
B. 

R escore . If an open-ended exercise 
~~ was scored under different condi- 
tions than presently held or if 
passage of time might affect scor- 
ing , responses from a previous 
assessment may be rescored at the 
same time that responses from a 
later assessment are scored. Re- 
sponses from an earlier assessment 
also may be held and not scored so 
that they can be sco r ed wi th r e- 
sponses from a later assessment. 

Respondent , A person who responds to 
the exercises in an assessment 
booklet. 

Response options . Different alterna- 
tives to a multiple-choice question 
that can be selected by the respon- 
dent. 

Review conferen ce. A conference held 
to review ERe objectives of a 
learning area to assure their 
acceptance as measures of the ob- 
jectives by scholars, educators and 
lay persons or to review exercises 
for racial, ethnic, social or re- 
gional bias. 

Sample . National Assessment does not 
assess an entire age popjl 3t ion but 
rather selects a representative 
sample from the age group to answer 
assessment items. (See Chapter 4 
for a description of National 
Assessment sampling procedures.) 

Scoring tjuid e. A guide for hand 
scoring an open-ended exercise that 
specifies descriptive or diagnostic 
categories by givinq definitions 
and sample responses. 

Scoring ovals. Scannable ovals 
printed beside multiple-choice op- 



tions and printed at the bottom of 
the page for open-ended exercises 
(to be used in hand scoring) . When 
ovals are marked, they can be 
scored by machine and responses 
recorded by computer. 

Sex . Results are reported for males 
and females. 

Size of community . Results are re- 
ported for four oize-of-community 
categories: big cities, fringes 
around big cities, medium cities 
and small places. Itiese categories 
are defined in Appendix B. 

SMSA. Standard Metropolitan Statis- 
tical Area. SMSAs are economic and 
social units defined by the U.S. 
Bureau of the Census. 

Standard error . A measure of sampling 
variability for a statistic. Be- 
cause of NAEP's complex sample de- 
sign, standard errors are estimated 
by jackknifing first-stage sample 
estimates. 

Stern . The portion of an exercise 
that states the problem or asks the 
question. 

Stimulus . For reading exercises, 
this is a visual stimulus used as 
part of the stem. 

Subject area . See learning area . 

Subpopulation or subgroup . Groups 
within the national population, 
such as males and females, for 
which data are reported. 

Supplementary principal's cjuestion- 
naire . A data collection form 
given to school officials. Ori this 
form, officials are asked to re- 
spond to quest ions concerning 
course offerings, materials and 
staffing specific to the learning 
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area being assessed. A sample of 
this questionnaire is found in Ap- 
pendix C. See also principal's 
questio n naire . 

Tapescript . A script prepared for 
the anno unce r to use in prod uc i ng 
the paced tape. It indicates ex- 
actly what is to be read or not 
read aloud to the students and in- 
dicates the amount of response time 
allowed for each exercise. See 
paced audio tape . 

Timing . Most NAEP exercises are ad- 
ministered with a paced audio tape 
to standardize data collection con- 
ditions. The tape includes the 
amount of time students are allowed 
to respond to each exercise. 

Type o f commun i ty . Results are re- 
ported for three type-of -community 
categories: disadvantaged urban, 
advantaged urban and rural. Defini- 



tions of these categories are found 
in Appendix B. 

User tape . See public-use data t ape. 

Variable . A classification of re- 
spondents. Standard reporting vari- 
ables are: region, sex, race, level 
of parental education, size of com- 
munity, type of community and grade 
in school. 

Weight . A multiplicative factor equal 
to the reciprocal of the probabil- 
ity of a respondent being selected 
for assessment with adjustment for 
nonresponse — an estimate of the 
number of persons in the population 
represented by a respondent in tb* 
sample. Theoretically, the sum of 
weights for all respondents at an 
age level is equal to the nimber of 
persons in the country at that age 
level. 
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10-W-25 The Third Assessment of Writing: 1978-79 

Released Exercise Set, February 1981 33.00 

10- W-40 Procedural Handbook: 1978-79 Writing 

Assessment, February 1981 7.40 

READING 

Twelve reports from the first assessment of reading 
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Assessments, October 1976 1.40 
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ADULTS (special probe; 
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Too? October 1979 1.00 
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